# Vespa documentation > Vespa is a powerful and scalable engine for low-latency computation over large data sets. It is designed for real-time applications that require a combination of search, recommendation, and personalization. Vespa allows developers to build applications that can handle high volumes of queries and data writes while maintaining fast response times. **Key Features** Vespa offers a rich set of features for building sophisticated search and recommendation applications: * **Real-time Indexing and Search:** Vespa provides low-latency CRUD (Create, Read, Update, Delete) operations, making data searchable in milliseconds after being fed. * **Approximate Nearest Neighbor (ANN) Search:** Vespa includes a highly efficient implementation of the HNSW algorithm for fast and accurate vector search, which can be combined with traditional filters. * **Flexible Ranking and Relevance:** Ranking is a first-class citizen in Vespa. It supports complex, multi-phase ranking expressions and can integrate with machine-learned models (e.g., ONNX, TensorFlow, XGBoost, LightGBM) to deliver highly relevant results. * **Scalability and Elasticity:** Vespa is designed to scale horizontally. Content clusters can be grown or shrunk on the fly without service interruptions, and data is automatically redistributed to maintain a balanced load. * **Rich Data Modeling:** Vespa supports a variety of data types, including structured data, unstructured text, and tensors for vector embeddings. It also supports parent-child relationships to model complex data hierarchies. * **Comprehensive Query Language:** Vespa's query language (YQL) allows for a combination of keyword search, structured filtering, and nearest neighbor search in a single query. * **Component-Based Architecture:** Vespa's container clusters host custom Java components (Searchers, Document Processors) that allow for extensive customization of query and data processing pipelines. **Architecture** A Vespa application consists of two main types of clusters: * **Stateless Container Clusters:** These clusters handle incoming queries and data writes. They host application components that process requests and responses, perform query rewriting, and federate to backend services. * **Stateful Content Clusters:** These clusters are responsible for storing, indexing, and searching data. They automatically manage data distribution and redundancy. Vespa's architecture is designed for high availability and fault tolerance. When a node fails, the system automatically re-routes traffic and re-distributes data to maintain service. **Application Package** A Vespa application is defined by an **application package**, which contains all the necessary configuration, schemas, components, and machine-learned models. This self-contained package allows for atomic deployments and ensures consistency between code and configuration. Key files in an application package include: * **`services.xml`**: Defines the services and clusters that make up the application, including their topology and resource allocation. * **`schemas/*.sd`**: Defines the document types, their fields, and how they should be indexed and searched. Rank profiles are also defined within schemas. **APIs and Interfaces** Vespa provides a comprehensive set of APIs for interacting with the system: * **Document API (`/document/v1/`)**: A REST API for performing CRUD operations on documents. * **Query API (`/search/`)**: A powerful API for querying data using YQL, with extensive options for ranking, grouping, and presentation. * **Configuration and Deployment APIs**: REST APIs for deploying application packages and managing system configuration. This overview provides a glimpse into the capabilities of the Vespa Search engine. For more in-depth information, please refer to each of the documentation links below. ## About Documentation ### About this documentation [](/en/learn/about-documentation.html.md "View as Markdown") #### About this documentation [](/en/learn/about-documentation.html.md "View as Markdown") The Vespa documentation ([https://docs.vespa.ai/](https://docs.vespa.ai/)) provides all the information required to use all Vespa features and deploy them in any supported environment. It is split into guides and tutorials, which explains features and how to use them to solve problems, and reference documentation which list complete information about all features and APIs. ##### Applicability The Vespa platform is open source, and can be deployed in self-managed systems and on the Vespa Cloud service. Some add-ons (but no core functionality) are only available under a commercial license. Documents that describe functionality with such limited applicability are clearly marked by one or more of the following chips: | Vespa Cloud This content is applicable to Vespa Cloud deployments. | Only applicable to Vespa Cloud deployments. | | Self-managed This content is applicable to self-managed Vespa systems. | Only applicable to self-managed deployments. | | Enterprise Not open source: This functionality is only available commercially. | Not open source: Available commercially only (both self-managed and on cloud unless also marked by one of the other chips above). | For clarity, any document _not_ marked with any of these chips describes functionality that is open source and available both on Vespa Cloud and self-managed deployments. ##### Contributing If you find errors or want to improve the documentation, [create an issue](https://github.com/vespa-engine/vespa/issues) or [contribute a fix](contributing). See the [README](https://docs.vespa.ai/README.md) before contributing. ##### Notation _Italic_ is used for: - Pathnames, filenames, program names, hostnames, and URLs - New terms where they are defined `Constant Width` is used for: - Programming language elements, code examples, keywords, functions, classes, interfaces, methods, etc. - Commands and command-line output Commands meant to be run on the command line are shown like this, prepended by a $ for the prompt: ``` $ export PATH=$VESPA_HOME/bin:$PATH #how to highlight text in \
```
Notes and other Important pieces of information are shown like:
**Note:** Some info here
**Important:** Important info here
**Warning:** Warning here
**Deprecated:** Deprecation warning here
Copyright © 2026 - [Cookie Preferences](#)
---
## Access Logging
### Access Logging
[](/en/operations/access-logging.html.md "View as Markdown")
#### Access Logging
[](/en/operations/access-logging.html.md "View as Markdown")
The Vespa access log format allows the logs to be processed by a number of available tools handling JSON based (log) files. With the ability to add custom key/value pairs to the log from any Searcher, you can easily track the decisions done by container components for given requests.
##### Vespa Access Log Format
In the Vespa access log, each log event is logged as a JSON object on a single line. The log format defines a list of fields that can be logged with every request. In addition to these fields, [custom key/value pairs](#logging-key-value-pairs-to-the-json-access-log-from-searchers) can be logged via Searcher code. Pre-defined fields:
| Name | Type | Description | Always present |
| --- | --- | --- | --- |
| ip | string | The IP address request came from | yes |
| time | number | UNIX timestamp with millisecond decimal precision (e.g. 1477828938.123) when request is received | yes |
| duration | number | The duration of the request in seconds with millisecond decimal precision (e.g. 0.123) | yes |
| responsesize | number | The size of the response in bytes | yes |
| code | number | The HTTP status code returned | yes |
| method | string | The HTTP method used (e.g. 'GET') | yes |
| uri | string | The request URI from path and beyond (e.g. '/search?query=test') | yes |
| version | string | The HTTP version (e.g. 'HTTP/1.1') | yes |
| agent | string | The user agent specified in the request | yes |
| host | string | The host header provided in the request | yes |
| scheme | string | The scheme of the request | yes |
| port | number | The IP port number of the interface on which the request was received | yes |
| remoteaddr | string | The IP address of the [remote client](#logging-remote-address-port) if specified in HTTP header | no |
| remoteport | string | The port used from the [remote client](#logging-remote-address-port) if specified in HTTP header | no |
| peeraddr | string | Address of immediate client making request if different from _remoteaddr_ | no |
| peerport | string | Port used by immediate client making request if different from _remoteport_ | no |
| user-principal | string | The name of the authenticated user (java.security.Principal.getName()) if principal is set | no |
| ssl-principal | string | The name of the x500 principal if client is authenticated through SSL/TLS | no |
| search | object | Object holding search specific fields | no |
| search.totalhits | number | The total number of hits for the query | no |
| search.hits | number | The hits returned in this specific response | no |
| search.coverage | object | Object holding [query coverage information](../performance/graceful-degradation.html) similar to that returned in result set. | no |
| connection | string | Reference to the connection log entry. See [Connection log](#connection-log) | no |
| attributes | object | Object holding [custom key/value pairs](#logging-key-value-pairs-to-the-json-access-log-from-searchers) logged in searcher. | no |
**Note:** IP addresses can be both IPv4 addresses in standard dotted format (e.g. 127.0.0.1) or IPv6 addresses in standard form with leading zeros omitted (e.g. 2222:1111:123:1234:0:0:0:4321).
An example log line will look like this (here, pretty-printed):
```json
{
"ip": "152.200.54.243",
"time": 920880005.023,
"duration": 0.122,
"responsesize": 9875,
"code": 200,
"method": "GET",
"uri": "/search?query=test¶m=value",
"version": "HTTP/1.1",
"agent": "Mozilla/4.05 [en] (Win95; I)",
"host": "localhost",
"search": {
"totalhits": 1234,
"hits": 0,
"coverage": {
"coverage": 98,
"documents": 100,
"degraded": {
"non-ideal-state": true
}
}
}
}
```
**Note:** The log format is extendable by design such that the order of the fields can be changed and new fields can be added between minor versions. Make sure any programmatic log handling is using a proper JSON processor.
Example: Decompress, pretty-print, with human-readable timestamps:
```
$[jq](https://stedolan.github.io/jq/)'. + {iso8601date:(.time | todateiso8601)}' \
<(unzstd -c /opt/vespa/logs/vespa/access/JsonAccessLog.default.20210601010000.zst)
```
###### Logging Remote Address/Port
In some cases when a request passes through an intermediate service, this service may add HTTP headers indicating the IP address and port of the real origin client. These values are logged as _remoteaddr_ and _remoteport_ respectively. Vespa will log the contents in any of the following HTTP request headers as _remoteaddr_: _X-Forwarded-For_, _Y-RA_, _YahooRemoteIP_ or _Client-IP_. If more than one of these headers are present, the precedence is in the order listed here, i.e. _X-Forwarded-For_ takes precedence over _Y-RA_. The contents of the _Y-RP_ HTTP request header will be logged as _remoteport_.
If the remote address or -port differs from those initiating the HTTP request, the address and port for the immediate client making the request are logged as _peeraddress_ and _peerport_ respectively.
##### Configuring Logging
**Note:** The `accesslog` element can only be overridden on self-hosted Vespa, not on Vespa Cloud.
For details on the access logging configuration see [accesslog in the container](../reference/applications/services/container.html#accesslog) element in _services.xml_.
Key configuration options include:
- **fileNamePattern**: Pattern for log file names with time variable support
- **rotationInterval**: Time-based rotation schedule (minutes since midnight)
- **rotationSize**: Size-based rotation threshold in bytes (0 = disabled)
- **rotationScheme**: Either 'sequence' or 'date'
- **compressionFormat**: GZIP or ZSTD compression for rotated files
###### Logging Request Content
Vespa supports logging of request content for specific URI paths. This is useful for inspecting query content of search POST requests or document operations of Document v1 POST/PUT requests. The request content is logged as a base64-encoded string in the JSON access log.
To configure request content logging, use the [request-content](../reference/applications/services/container.html#request-content) element in the accesslog configuration in _services.xml_.
Here is an example of how the request content appears in the JSON access log:
```
{
...
"method": "POST",
"uri": "/search",
...,
"request-content": {
"type": "application/json; charset=utf-8",
"length": 12345,
"body": ""
}
}
```
###### File name pattern
The file name pattern is expanded using the time when the file is created. The following parts in the file name are expanded:
| Field | Format | Meaning | Example |
| --- | --- | --- | --- |
| %Y | YYYY | Year | 2003 |
| %m | MM | Month, numeric | 08 |
| %x | MMM | Month, textual | Aug |
| %d | dd | Date | 25 |
| %H | HH | Hour | 14 |
| %M | mm | Minute | 30 |
| %S | ss | Seconds | 35 |
| %s | SSS | Milliseconds | 123 |
| %Z | Z | Time zone | -0400 |
| %T | Long | System.currentTimeMillis | 1349333576093 |
| %% | % | Escape percentage | % |
##### Log rotation
Apache httpd style log _rotation_ can be configured by setting the _rotationScheme_. There's two alternatives for the rotationScheme, sequence and date. Rotation can be triggered by time intervals using _rotationInterval_ and/or by file size using _rotationSize_.
###### Sequence rotation scheme
The _fileNamePattern_ is used for the active log file name (which in this case will often be a constant string). At rotation, this file is given the name fileNamePattern.N where N is 1 + the largest integer found by extracting the integers from all files ending by .\ in the same directory
```xml
```
###### Date rotation scheme
The _fileNamePattern_ is used for the active log file name here too, but the log files are not renamed at rotation. Instead, you must specify a time-dependent fileNamePattern so that each time a new log file is created, the name is unique. In addition, a symlink is created pointing to the active log file. The name of the symlink is specified using _symlinkName_.
```xml
```
###### Rotation interval
The time of rotation is controlled by setting _rotationInterval_:
```xml
```
The rotationInterval is a list of numbers specifying when to do rotation. Each element represents the number of minutes since midnight. Ending the list with '...' means continuing the [arithmetic progression](https://en.wikipedia.org/wiki/Arithmetic_progression) defined by the two last numbers for the rest of the day. E.g. "0 100 240 480 ..." is expanded to "0 100 240 480 720 960 1200"
###### Log retention
Access logs are rotated, but not deleted by Vespa processes. It is up to the application owner to take care of archiving of access logs.
##### Logging Key/Value pairs to the JSON Access Log from Searchers
To add a key/value pair to the access log from a searcher, use
```
query/result.getContext(true).logValue(key,value)
```
Such key/value pairs may be added from any thread participating in handling the query without incurring synchronization overhead.
If the same key is logged multiple times, the values written will be included in the log as an array of strings rather than a single string value.
The key/value pairs are added to the _attributes_ object in the log.
An example log line will then look something like this:
```json
{"ip":"152.200.54.243","time":920880005.023,"duration":0.122,"responsesize":9875,"code":200,"method":"GET","uri":"/search?query=test¶m=value","version":"HTTP/1.1","agent":"Mozilla/4.05 [en] (Win95; I)","host":"localhost","search":{"totalhits":1234,"hits":0},"attributes":{"singlevalue":"value1","multivalue":["value2","value3"]}}
```
A pretty print version of the same example:
```json
{
"ip": "152.200.54.243",
"time": 920880005.023,
"duration": 0.122,
"responsesize": 9875,
"code": 200,
"method": "GET",
"uri": "/search?query=test¶m=value",
"version": "HTTP/1.1",
"agent": "Mozilla/4.05 [en] (Win95; I)",
"host": "localhost",
"search": {
"totalhits": 1234,
"hits": 0
},
"attributes": {
"singlevalue": "value1",
"multivalue": [
"value2",
"value3"
]
}
}
```
##### Connection log
In addition to the access log, one entry per connection is written to the connection log. This entry is written on connection close. Available fields:
| Name | Type | Description | Always present |
| --- | --- | --- | --- |
| id | string | Unique ID of the connection, referenced from access log. | yes |
| timestamp | number | Timestamp (ISO8601 format) when the connection was opened | yes |
| duration | number | The duration of the request in seconds with millisecond decimal precision (e.g. 0.123) | yes |
| peerAddress | string | IP address used by immediate client making request | yes |
| peerPort | number | Port used by immediate client making request | yes |
| localAddress | string | The local IP address the request was received on | yes |
| localPort | number | The local port the request was received on | yes |
| remoteAddress | string | Original client ip, if proxy protocol enabled | no |
| remotePort | number | Original client port, if proxy protocol enabled | no |
| httpBytesReceived | number | Number of HTTP bytes sent over the connection | no |
| httpBytesSent | number | Number of HTTP bytes received over the connection | no |
| requests | number | Number of requests sent by the client | no |
| responses | number | Number of responses sent to the client | no |
| ssl | object | Detailed information on ssl connection | no |
##### SSL information
| Name | Type | Description | Always present |
| --- | --- | --- | --- |
| clientSubject | string | Client certificate subject | no |
| clientNotBefore | string | Client certificate valid from | no |
| clientNotAfter | string | Client certificate valid to | no |
| sessionId | string | SSL session id | no |
| protocol | string | SSL protocol | no |
| cipherSuite | string | Name of session cipher suite | no |
| sniServerName | string | SNI server name | no |
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Access Logging](#page-title)
- [Vespa Access Log Format](#access-log-format)
- [Logging Remote Address/Port](#logging-remote-address-port)
- [Configuring Logging](#configuring-logging)
- [Logging Request Content](#logging-request-content)
- [File name pattern](#file-name-pattern)
- [Log rotation](#log-rotation)
- [Sequence rotation scheme](#sequence-rotation-scheme)
- [Date rotation scheme](#date-rotation-scheme)
- [Rotation interval](#rotation-interval)
- [Log retention](#log-retention)
- [Logging Key/Value pairs to the JSON Access Log from Searchers](#logging-key-value-pairs-to-the-json-access-log-from-searchers)
- [Connection log](#connection-log)
- [SSL information](#ssl-information)
---
## Self Managed
### Administrative Procedures
[](/en/operations/self-managed/admin-procedures.html.md "View as Markdown")
#### Administrative Procedures
[](/en/operations/self-managed/admin-procedures.html.md "View as Markdown")
##### Install
Refer to the [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) sample application for a primer on how to set up a cluster - use this as a starting point. Try the [Multinode testing and observability](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode) sample app to get familiar with interfaces and behavior.
##### Vespa start / stop / restart
Start and stop all services on a node:
```
$ $VESPA_HOME/bin/[vespa-start-services](../../reference/operations/self-managed/tools.html#vespa-start-services)$ $VESPA_HOME/bin/[vespa-stop-services](../../reference/operations/self-managed/tools.html#vespa-stop-services)
```
Likewise, for the config server:
```
$ $VESPA_HOME/bin/[vespa-start-configserver](../../reference/operations/self-managed/tools.html#vespa-start-configserver)$ $VESPA_HOME/bin/[vespa-stop-configserver](../../reference/operations/self-managed/tools.html#vespa-stop-configserver)
```
There is no _restart_ command, do a _stop_ then _start_ for a restart. Learn more about which processes / services are started at [Vespa startup](config-sentinel.html), read the [start sequence](configuration-server.html#start-sequence) and find training videos in the vespaengine [YouTube channel](https://www.youtube.com/@vespaai).
Use [vespa-sentinel-cmd](../../reference/operations/self-managed/tools.html#vespa-sentinel-cmd) to stop/start individual services.
**Important:** Running _vespa-stop-services_ on a content node will call [prepareRestart](../../reference/operations/self-managed/tools.html#vespa-proton-cmd) to optimize restart time, and is the recommended way to stop Vespa on a node.
See [multinode](multinode-systems.html#aws-ec2) for _systemd_ /_systemctl_ examples. [Docker containers](docker-containers.html) has relevant start/stop information, too.
###### Content node maintenance mode
When stopping a content node _temporarily_ (e.g. for a software upgrade), consider manually setting the node into [maintenance mode](../../reference/api/cluster-v2.html#maintenance) _before_ stopping the node to prevent automatic redistribution of data while the node is down. Maintenance mode must be manually removed once the node has come back online. See also: [cluster state](#cluster-state).
Example of setting a node with [distribution key](../../reference/applications/services/content.html#node) 42 into `maintenance` mode using [vespa-set-node-state](../../reference/operations/self-managed/tools.html#vespa-set-node-state), additionally supplying a reason that will be recorded by the cluster controller:
```
$ vespa-set-node-state --type storage --index 42 maintenance "rebooting for software upgrade"
```
After the node has come back online, clear maintenance mode by marking the node as `up`:
```
$ vespa-set-node-state --type storage --index 42 up
```
Note that if the above commands are executed _locally_ on the host running the services for node 42, `--index 42` can be omitted; `vespa-set-node-state` will use the distribution key of the local node if no `--index` has been explicitly specified.
##### System status
- Use [vespa-config-status](../../reference/operations/self-managed/tools.html#vespa-config-status) on a node in [hosts.xml](../../reference/applications/hosts.html) to verify all services run with updated config
- Make sure [VESPA\_CONFIGSERVERS](files-processes-and-ports.html#environment-variables) is set and identical on all nodes in hosts.xml
- Use the _cluster controller_ status page (below) to track the status of search/storage nodes.
- Check [logs](../../reference/operations/log-files.html)
- Use performance graphs, System Activity Report (_sar_) or [status pages](#status-pages) to track load
- Use [query tracing](../../reference/api/query.html#trace.level)
- Disk and/or memory might be exhausted and block feeding - recover from [feed block](/en/writing/feed-block.html)
##### Status pages
All Vespa services have status pages, for showing health, Vespa version, config, and metrics. Status pages are subject to change at any time - take care when automating. Procedure:
1. **Find the port:** The status pages runs on ports assigned by Vespa. To find status page ports, use [vespa-model-inspect](../../reference/operations/self-managed/tools.html#vespa-model-inspect) to list the services run in the application.
```
$ vespa-model-inspect services
```
To find the status page port for a specific node for a specific service, pick the correct service and run:
```
$ vespa-model-inspect service [Options]
```
2. **Get the status and metrics:**_distributor_, _storagenode_, _searchnode_ and _container-clustercontroller_ are content services with status pages. These ports are tagged HTTP. The cluster controller have multiple ports tagged HTTP, where the port tagged STATE is the one with the status page. Try connecting to the root at the port, or /state/v1/metrics. The _distributor_ and _storagenode_ status pages are available at `/`:
```
$ vespa-model-inspect service searchnode
searchnode @ myhost.mydomain.com : search
search/search/cluster.search/0
tcp/myhost.mydomain.com:19110 (STATUS ADMIN RTC RPC)
tcp/myhost.mydomain.com:19111 (FS4)
tcp/myhost.mydomain.com:19112 (TEST HACK SRMP)
tcp/myhost.mydomain.com:19113 (ENGINES-PROVIDER RPC)tcp/myhost.mydomain.com:19114 (HEALTH JSON HTTP)$ curl http://myhost.mydomain.com:19114/state/v1/metrics
...
$ vespa-model-inspect service distributor
distributor @ myhost.mydomain.com : content
search/distributor/0
tcp/myhost.mydomain.com:19116 (MESSAGING)
tcp/myhost.mydomain.com:19117 (STATUS RPC)tcp/myhost.mydomain.com:19118 (STATE STATUS HTTP)$ curl http://myhost.mydomain.com:19118/state/v1/metrics
...
$ curl http://myhost.mydomain.com:19118/
...
```
3. **Use the cluster controller status page**: A status page for the cluster controller is available at the status port at `http://hostname:port/clustercontroller-status/v1/`. If _clustername_ is not specified, the available clusters will be listed. The cluster controller leader status page will show if any nodes are operating with differing cluster state versions. It will also show how many data buckets are pending merging (document set reconciliation) due to either missing or being out of sync.
```
$[vespa-model-inspect](../../reference/operations/self-managed/tools.html#vespa-model-inspect)service container-clustercontroller | grep HTTP
```
With multiple cluster controllers, look at the one with a "/0" suffix in its config ID; it is the preferred leader.
The cluster state version is listed under the _SSV_ table column. Divergence here usually points to host or networking issues.
##### Cluster state
Cluster and node state information is available through the [/cluster/v2 API](../../reference/api/cluster-v2.html). This API can also be used to set a _user state_ for a node - alternatively use:
- [vespa-get-cluster-state](../../reference/operations/self-managed/tools.html#vespa-get-cluster-state)
- [vespa-get-node-state](../../reference/operations/self-managed/tools.html#vespa-get-node-state)
- [vespa-set-node-state](../../reference/operations/self-managed/tools.html#vespa-set-node-state)
Also see the cluster controller [status page](#status-pages).
State is persisted in a ZooKeeper cluster, restarting/changing a cluster controller preserves:
- Last cluster state version number, for new cluster controller handover at restarts
- User states, set by operators - i.e. nodes manually set to down / maintenance
In case of state data lost, the cluster state is reset - see [cluster controller](../../content/content-nodes.html#cluster-controller) for implications.
##### Cluster controller configuration
It is recommended to run cluster controllers on the same hosts as [config servers](configuration-server.html), as they share a zookeeper cluster for state and deploying three nodes is best practise for both. See the [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) sample app for a working example.
To configure the cluster controller, use [services.xml](../../reference/applications/services/content.html#cluster-controller) and/or add [configuration](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/fleetcontroller.def) under the _services_ element - example:
```xml
5000
```
A broken content node may end up with processes constantly restarting. It may die during initialization due to accessing corrupt files, or it may die when it starts receiving requests of a given type triggering a node local bug. This is bad for distributor nodes, as these restarts create constant ownership transfer between distributors, causing windows where buckets are unavailable.
The cluster controller has functionality for detecting such nodes. If a node restarts in a way that is not detected as a controlled shutdown, more than [max\_premature\_crashes](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/fleetcontroller.def), the cluster controller will set the wanted state of this node to be down.
Detecting a controlled restart is currently a bit tricky. A controlled restart is typically initiated by sending a TERM signal to the process. Not having any other sign, the content layer has to assume that all TERM signals are the cause of controlled shutdowns. Thus, if the process keep being killed by kernel due to using too much memory, this will look like controlled shutdowns to the content layer.
##### Monitor distance to ideal state
Refer to the [distribution algorithm](../../content/idealstate.html). Use distributor [status pages](#status-pages) to inspect state metrics, see [metrics](../../content/content-nodes.html#metrics). `idealstate.merge_bucket.pending` is the best metric to track, it is 0 when the cluster is balanced - a non-zero value indicates buckets out of sync.
##### Cluster configuration
- Running `vespa prepare` will not change served configuration until `vespa activate` is run. `vespa prepare` will warn about all config changes that require restart.
- Refer to [schemas](../../basics/schemas.html) for how to add/change/remove these.
- Refer to [elasticity](../../content/elasticity.html) for how to add/remove capacity from a Vespa cluster, procedure below.
- See [chained components](../../applications/chaining.html) for how to add or remove searchers and document processors.
- Refer to the [sizing examples](sizing-examples.html) for changing from a _flat_ to _grouped_ content cluster.
##### Add or remove a content node
1. **Node setup:** Prepare the node by installing software, set up the file systems/directories and set [VESPA\_CONFIGSERVERS](files-processes-and-ports.html#environment-variables). [Start](#vespa-start-stop-restart) the node.
2. **Modify configuration:** Add/remove a [node](../../reference/applications/services/content.html#node)-element in _services.xml_ and [hosts.xml](../../reference/applications/hosts.html). Refer to [multinode install](multinode-systems.html). Make sure the _distribution-key_ is unique.
3. **Deploy**: [Observe metrics](#monitor-distance-to-ideal-state) to track progress as the cluster redistributes documents. Use the [cluster controller](../../content/content-nodes.html#cluster-controller) to monitor the state of the cluster.
4. **Tune performance (optional):** Use [maxpendingidealstateoperations](https://github.com/vespa-engine/vespa/blob/master/storage/src/vespa/storage/config/stor-distributormanager.def) to tune concurrency of bucket merge operations from distributor nodes. Likewise, tune [merges](../../reference/applications/services/content.html#merges) - concurrent merge operations per content node. The tradeoff is speed of bucket replication vs use of resources, which impacts the applications' regular load.
5. **Finish:** The cluster is done redistributing when `idealstate.merge_bucket.pending` is zero on all distributors.
Do not remove more than _redundancy_-1 nodes at a time, to avoid data loss. Observe `idealstate.merge_bucket.pending` to know bucket replica status, when zero on all distributor nodes, it is safe to remove more nodes. If [grouped distribution](../../content/elasticity.html#grouped-distribution) is used to control bucket replicas, remove all nodes in a group if the redundancy settings ensure replicas in each group.
To increase bucket redundancy level before taking nodes out, [retire](../../content/content-nodes.html) nodes. Again, track `idealstate.merge_bucket.pending` to know when done. Use the [/cluster/v2 API](../../reference/api/cluster-v2.html) or [vespa-set-node-state](../../reference/operations/self-managed/tools.html#vespa-set-node-state) to set a node to the _retired_ state. You can set any number of nodes retired at the same time. The [cluster controller's](../../content/content-nodes.html#cluster-controller) status page lists node states.
An alternative to increasing cluster size is building a new cluster, then migrate documents to it. This is supported using [visiting](../../writing/visiting.html).
To _merge_ two content clusters, add nodes to the cluster like above, considering:
- [distribution-keys](../../reference/applications/services/content.html#node) must be unique. Modify paths like _$VESPA\_HOME/var/db/vespa/search/mycluster/n3_ before adding the node.
- Set [VESPA\_CONFIGSERVERS](files-processes-and-ports.html#environment-variables), then start the node.
##### Topology change
Read [changing topology first](../../content/elasticity.html#changing-topology), and plan the sequence of steps.
Make sure to not change the `distribution-key` for nodes in _services.xml_.
It is not required to restart nodes as part of this process
##### Add or remove services on a node
It is possible to run multiple Vespa services on the same host. If changing the services on a given host, stop Vespa on the given host before running `vespa activate`. This is because the services are dynamically allocated port numbers, depending on what is running on the host. Consider if some of the services changed are used by services on other hosts. In that case, restart services on those hosts too. Procedure:
1. Edit _services.xml_ and _hosts.xml_
2. Stop Vespa on the nodes that have changes
3. Run `vespa prepare` and `vespa activate`
4. Start Vespa on the nodes that have changes
##### Troubleshooting
Also see the [FAQ](../../learn/faq).
| No endpoint |
Most problems with the quick start guides are due to Docker out of memory. Make sure at least 6G memory is allocated to Docker:
```
$ docker info | grep "Total Memory"
or
$ podman info | grep "memTotal"
```
OOM symptoms include
```
INFO: Problem with Handshake localhost:8080 ssl=false: localhost:8080 failed to respond
```
The container is named _vespa_ in the guides, for a shell do:
```
$ docker exec -it vespa bash
```
|
| Log viewing |
Use [vespa-logfmt](../../reference/operations/self-managed/tools.html#vespa-logfmt) to view the vespa log - example:
```
$ /opt/vespa/bin/vespa-logfmt -l warning,error
```
|
| Json |
For json pretty-print, append
```
| python -m json.tool
```
to commands that output json - or use [jq](https://stedolan.github.io/jq/). |
| Routing |
Vespa lets application set up custom document processing / indexing, with different feed endpoints. Refer to [indexing](../../writing/indexing.html) for how to configure this in _services.xml_.
[#13193](https://github.com/vespa-engine/vespa/issues/13193) has a summary of problems and solutions.
|
| Tracing |
Use [tracelevel](../../reference/api/document-v1.html#request-parameters) to dump the routes and hops for a write operation - example:
```
$ curl -H Content-Type:application/json --data-binary @docs.json \
$ENDPOINT/document/v1/mynamespace/doc/docid/1?tracelevel=4 | jq .
{
"pathId": "/document/v1/mynamespace/doc/docid/1",
"id": "id:mynamespace:doc::1",
"trace": [
{ "message": "[1623413878.905] Sending message (version 7.418.23) from client to ..." },
{ "message": "[1623413878.906] Message (type 100004) received at 'default/container.0' ..." },
{ "message": "[1623413878.907] Sending message (version 7.418.23) from 'default/container.0' ..." },
{ "message": "[1623413878.907] Message (type 100004) received at 'default/container.0' ..." },
{ "message": "[1623413878.909] Selecting route" },
{ "message": "[1623413878.909] No cluster state cached. Sending to random distributor." }
```
|
##### Clean start mode
There has been rare occasions were Vespa stored data that was internally inconsistent. For those circumstances it is possible to start the node in a [validate\_and\_sanitize\_docstore](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/proton.def) mode. This will do its best to clean up inconsistent data. However, detecting that this is required is not easy, consult the Vespa Team first. In order for this approach to work, all nodes must be stopped before enabling this feature - this to make sure the data is not redistributed.
##### Content cluster configuration
| Availability vs resources |
Keeping index structures costs resources. Not all replicas of buckets are necessarily searchable, unless configured using [searchable-copies](../../reference/applications/services/content.html#searchable-copies). As Vespa indexes buckets on-demand, the most cost-efficient setting is 1, if one can tolerate temporary coverage loss during node failures.
|
| Data retention vs size |
When a document is removed, the document data is not immediately purged. Instead, _remove-entries_ (tombstones of removed documents) are kept for a configurable amount of time. The default is two weeks, refer to [removed-db prune age](../../reference/applications/services/content.html#removed-db-prune-age). This ensures that removed documents stay removed in a distributed system where nodes change state. Entries are removed periodically after expiry. Hence, if a node comes back up after being down for more than two weeks, removed documents are available again, unless the data on the node is wiped first. A larger _prune age_ will grow the storage size as this keeps document and tombstones longer.
**Note:** The backend does not store remove-entries for nonexistent documents. This to prevent clients sending wrong document identifiers from filling a cluster with invalid remove-entries. A side effect is that if a problem has caused all replicas of a bucket to be unavailable, documents in this bucket cannot be marked removed until at least one replica is available again. Documents are written in new bucket replicas while the others are down - if these are removed, then older versions of these will not re-emerge, as the most recent change wins.
|
| Transition time |
See [transition-time](../../reference/applications/services/content.html#transition-time) for tradeoffs for how quickly nodes are set down vs. system stability.
|
| Removing unstable nodes |
One can configure how many times a node is allowed to crash before it will automatically be removed. The crash count is reset if the node has been up or down continuously for more than the [stable state period](../../reference/applications/services/content.html#stable-state-period). If the crash count exceeds [max premature crashes](../../reference/applications/services/content.html#max-premature-crashes), the node will be disabled. Refer to [troubleshooting](#troubleshooting).
|
| Minimal amount of nodes required to be available |
A cluster is typically sized to handle a given load. A given percentage of the cluster resources are required for normal operations, and the remainder is the available resources that can be used if some of the nodes are no longer usable. If the cluster loses enough nodes, it will be overloaded:
- Remaining nodes may create disk full situation. This will likely fail a lot of write operations, and if disk is shared with OS, it may also stop the node from functioning.
- Partition queues will grow to maximum size. As queues are processed in FIFO order, operations are likely to get long latencies.
- Many operations may time out while being processed, causing the operation to be resent, adding more load to the cluster.
- When new nodes are added, they cannot serve requests before data is moved to the new nodes from the already overloaded nodes. Moving data puts even more load on the existing nodes, and as moving data is typically not high priority this may never actually happen.
To configure what the minimal cluster size is, use [min-distributor-up-ratio](../../reference/applications/services/content.html#min-distributor-up-ratio) and [min-storage-up-ratio](../../reference/applications/services/content.html#min-storage-up-ratio). |
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Administrative Procedures](#page-title)
- [Install](#install)
- [Vespa start / stop / restart](#vespa-start-stop-restart)
- [Content node maintenance mode](#content-node-maintenance-mode)
- [System status](#system-status)
- [Status pages](#status-pages)
- [Cluster state](#cluster-state)
- [Cluster controller configuration](#cluster-controller-configuration)
- [Monitor distance to ideal state](#monitor-distance-to-ideal-state)
- [Cluster configuration](#cluster-configuration)
- [Add or remove a content node](#add-or-remove-a-content-node)
- [Topology change](#topology-change)
- [Add or remove services on a node](#add-or-remove-services-on-a-node)
- [Troubleshooting](#troubleshooting)
- [Clean start mode](#clean-start-mode)
- [Content cluster configuration](#content-cluster-configuration)
---
### Build / install Vespa
[](/en/operations/self-managed/build-install.html.md "View as Markdown")
#### Build / install Vespa
[](/en/operations/self-managed/build-install.html.md "View as Markdown")
To develop with Vespa, follow the [guide](https://github.com/vespa-engine/vespa#building) to set up a development environment on AlmaLinux 8 using Docker.
Build Vespa Java artifacts with Java \>= 17 and Maven \>= 3.6.3. Once built, Vespa Java artifacts are ready to be used and one can build a Vespa application using the [bundle plugin](../../applications/bundles.html#maven-bundle-plugin).
```
$ export MAVEN_OPTS="-Xms128m -Xmx1024m"
$ ./bootstrap.sh java && mvn install
```
See [vespa.ai releases](../../learn/releases.html).
##### Container images
| Image | Description |
| --- | --- |
| [docker.io/vespaengine/vespa](https://hub.docker.com/r/vespaengine/vespa)
[ghcr.io/vespa-engine/vespa](https://github.com/orgs/vespa-engine/packages/container/package/vespa) | Container image for running Vespa. |
| [docker.io/vespaengine/vespa-build-almalinux-8](https://hub.docker.com/r/vespaengine/vespa-build-almalinux-8) | Container image for building Vespa on AlmaLinux 8. |
| [docker.io/vespaengine/vespa-dev-almalinux-8](https://hub.docker.com/r/vespaengine/vespa-dev-almalinux-8) | Container image for development of Vespa on AlmaLinux 8. Used for incremental building and system testing. |
##### RPMs
Dependency graph:

Installing Vespa on AlmaLinux 8:
```
$ dnf config-manager \
--add-repo https://raw.githubusercontent.com/vespa-engine/vespa/master/dist/vespa-engine.repo
$ dnf config-manager --enable powertools
$ dnf install -y epel-release
$ dnf install -y vespa
```
Package repository hosting is graciously provided by [Cloudsmith](https://cloudsmith.com) which is a fully hosted, cloud-native and universal package management solution: [](https://cloudsmith.com)
**Important:** Please note that the retention of released RPMs in the repository is limited to the latest 50 releases. Use the Docker images (above) for installations of specific versions older than this. Any problems with released rpm packages will be fixed in subsequent releases, please [report any issues](https://vespa.ai/support/) - troubleshoot using the [install example](/en/operations/self-managed/multinode-systems.html#aws-ec2-singlenode).
Refer to [vespa.spec](https://github.com/vespa-engine/vespa/blob/master/dist/vespa.spec). Build RPMs for a given Vespa version X.Y.Z:
```
$ git clone https://github.com/vespa-engine/vespa
$ cd vespa
$ git checkout vX.Y.Z
$ docker run --rm -ti -v $(pwd):/wd:Z -w /wd \
docker.io/vespaengine/vespa-build-almalinux-8:latest \
make -f .copr/Makefile rpms outdir=/wd
$ ls *.rpm | grep -v debug
vespa-8.781.11-1.el8.src.rpm
vespa-8.781.11-1.el8.x86_64.rpm
vespa-ann-benchmark-8.781.11-1.el8.x86_64.rpm
vespa-base-8.781.11-1.el8.x86_64.rpm
vespa-base-libs-8.781.11-1.el8.x86_64.rpm
vespa-clients-8.781.11-1.el8.x86_64.rpm
vespa-config-model-fat-8.781.11-1.el8.x86_64.rpm
vespa-jars-8.781.11-1.el8.x86_64.rpm
vespa-libs-8.781.11.el8.x86_64.rpm
vespa-malloc-8.781.11-1.el8.x86_64.rpm
vespa-node-admin-8.781.11-1.el8.x86_64.rpm
vespa-tools-8.781.11-1.el8.x86_64.rpm
```
Find most utilities in the vespa-x.y.z\*.rpm - other RPMs:
| RPM | Description |
| --- | --- |
| vespa-tools | Tools accessing Vespa endpoints for query or document operations:
- [vespa-destination](/en/reference/operations/self-managed/tools.html#vespa-destination)
- [vespa-fbench](/en/reference/operations/tools.html#vespa-fbench)
- [vespa-feeder](/en/reference/operations/self-managed/tools.html#vespa-feeder)
- [vespa-get](/en/reference/operations/self-managed/tools.html#vespa-get)
- [vespa-query-profile-dump-tool](/en/reference/operations/tools.html#vespa-query-profile-dump-tool)
- [vespa-stat](/en/reference/operations/self-managed/tools.html#vespa-stat)
- [vespa-summary-benchmark](/en/reference/operations/self-managed/tools.html#vespa-summary-benchmark)
- [vespa-visit](/en/reference/operations/self-managed/tools.html#vespa-visit)
- [vespa-visit-target](/en/reference/operations/self-managed/tools.html#vespa-visit-target)
|
| vespa-malloc | Vespa has its own memory allocator, _vespa-malloc_ - refer to _/opt/vespa/etc/vespamalloc.conf_ |
| vespa-clients | _vespa-feed-client.jar_ - see [vespa-feed-client](../../clients/vespa-feed-client.html) |
Copyright © 2026 - [Cookie Preferences](#)
---
### Configuration proxy
[](/en/operations/self-managed/config-proxy.html.md "View as Markdown")
#### Configuration proxy
[](/en/operations/self-managed/config-proxy.html.md "View as Markdown")
Read [application packages](../../basics/applications.html) for an overview of the cloud config system. The _config proxy_ runs on every Vespa node. It has a set of config sources, defined in [VESPA\_CONFIGSERVERS](files-processes-and-ports.html#environment-variables).
The config proxy will act as a proxy for config clients on the same machine, so that all clients can ask for config on _localhost:19090_. The _config source_ that the config proxy uses is set in [VESPA\_CONFIGSERVERS](files-processes-and-ports.html#environment-variables) and consists of one or more config sources (the addresses of [config servers](configuration-server.html)).
The proxy has a memory cache that is used to serve configs if it is possible. In default mode, the proxy will have an outstanding request to the config server that will return when the config has changed (a new generation of config). This means that every time config changes on the config server, the proxy will get a response, update its cache and respond to all its clients with the changed config.
The config proxy has two modes:
| Mode | Description |
| --- | --- |
| default | Gets config from server and stores in memory cache. The config proxy will always be started in _default_ mode. Serves from cache if possible. Always uses a config source. If restarted, it will lose all configs that were cached in memory. |
| memorycache | Serves config from memory cache only. Never uses a config source. A restart will lose all cached configs. Setting the mode to _memorycache_ will make all applications on the node work as before (given that they have previously been running and requested config), since the config proxy will serve config from cache and work without connection to any config server. Applications on this node will not work if the config proxy stops, is restarted or crashes. |
Use [vespa-configproxy-cmd](../../reference/operations/self-managed/tools.html#vespa-configproxy-cmd) to inspect cached configs, mode, config sources etc., there are also some commands to change some of the settings. Run the command as:
```
$ vespa-configproxy-cmd -m
```
to see all possible commands.
##### Detaching from config servers
```
$ vespa-configproxy-cmd -m setmode memorycache
```
##### Inspecting config
To inspect the configuration for a service, in this example a searchnode (proton) instance, do:
1. Find the active config generation used by the service, using [/state/v1/config](../../reference/api/state-v1.html#state-v1-config) - example for _http://localhost:19110/state/v1/config_, here the generation is 2:
```json
{
"config": {
"generation": 2,
"proton": {
"generation": 2
},
"proton.documentdb.music": {
"generation": 2
}
}
}
```
2. Find the relevant _config definition name_, _config id_ and _config generation_ using [vespa-configproxy-cmd](../../reference/operations/self-managed/tools.html#vespa-configproxy-cmd) - e.g.:
```
$ vespa-configproxy-cmd | grep protonvespa.config.search.core.proton,music/search/cluster.music/0,2,MD5:40087d6195cedb1840721b55eb333735,XXHASH64:43829e79cea8e714
```
`vespa.config.search.core.proton` is the _config definition name_ for this particular config, `music/search/cluster.music/0` is the _config id_ used by the proton service instance on this node and `2` is the active config generation. This means, the service is using the correct config generation as it is matching the /state/v1/config response (a restart can be required for some config changes).
3. Get the generated config using [vespa-get-config](../../reference/operations/self-managed/tools.html#vespa-get-config) - e.g.:
```
$ vespa-get-config -n vespa.config.search.core.proton -i music/search/cluster.music/0
basedir "/opt/vespa/var/db/vespa/search/cluster.music/n0"
rpcport 19106
httpport 19110
...
```
**Important:** Omitting `-i` will return the default configuration, meaning not generated for the active service instance.
Copyright © 2026 - [Cookie Preferences](#)
---
### Config sentinel
[](/en/operations/self-managed/config-sentinel.html.md "View as Markdown")
#### Config sentinel
[](/en/operations/self-managed/config-sentinel.html.md "View as Markdown")
The config sentinel starts and stops services - and restart failed services unless they are manually stopped. All nodes in a Vespa system have at least these running processes:
| Process | Description |
| --- | --- |
| [config-proxy](config-proxy.html) | Proxies config requests between Vespa applications and the configserver node. All configuration is cached locally so that this node can maintain its current configuration, even if the configserver shuts down. |
| config-sentinel | Registers itself with the _config-proxy_ and subscribes to and enforces node configuration, meaning the configuration of what services should be run locally, and with what parameters. |
| [vespa-logd](../../reference/operations/log-files.html#logd) | Monitors _$VESPA\_HOME/logs/vespa/vespa.log_, which is used by all other services, and relays everything to the [log-server](../../reference/operations/log-files.html#log-server). |
| [metrics-proxy](monitoring.html#metrics-proxy) | Provides APIs for metrics access to all nodes and services. |

Start sequence:
1. _config server(s)_ are started and application config is deployed to them - see [config server operations](configuration-server.html).
2. _config-proxy_ is started. The environment variables [VESPA\_CONFIGSERVERS](files-processes-and-ports.html#environment-variables) and [VESPA\_CONFIGSERVER\_RPC\_PORT](files-processes-and-ports.html#environment-variables) are used to connect to the [config-server(s)](configuration-server.html). It will retry all config servers in case some are down.
3. _config-sentinel_ is started, and subscribes to node configuration (i.e. a service list) from _config-proxy_ using its hostname as the [config id](../../applications/configapi-dev.html#config-id). See [Node and network setup](node-setup.html) for details about how the hostname is detected and how to override it. The config for the config-sentinel (the service list) lists the processes to be started, along with the _config id_ to assign to each, typically the logical name of that service instance.
4. _config-proxy_ subscribes to node configuration from _config-server_, caches it, and returns the result to _config-sentinel_
5. _config-sentinel_ starts the services given in the node configuration, with the config id as argument. See example output below, like _id="search/qrservers/qrserver.0"_. _logd_ and _metrics-proxy_ are always started, regardless of configuration. Each service:
1. Subscribes to configuration from _config-proxy_.
2. _config-proxy_ subscribes to configuration from _config-server_, caches it and returns result to the service.
3. The service runs according to its configuration, logging to _$VESPA\_HOME/logs/vespa/vespa.log_. The processes instantiate internal components, each assigned the same or another config id, and instantiating further components.
Also see [cluster startup](#cluster-startup) for a minimum nodes-up start setting.
When new config is deployed to _config-servers_ they propagate the changed configuration to nodes subscribing to it. In turn, these nodes reconfigure themselves accordingly.
##### User interface
The config sentinel runs an RPC service which can be used to list, start and stop the services supposed to run on that node. This can be useful for testing and debugging. Use [vespa-sentinel-cmd](../../reference/operations/self-managed/tools.html#vespa-sentinel-cmd) to trigger these actions. Example output from `vespa-sentinel-cmd list`:
```
vespa-sentinel-cmd 'sentinel.ls' OK.
container state=RUNNING mode=AUTO pid=27993 exitstatus=0 id="default/container.0"
container-clustercontroller state=RUNNING mode=AUTO pid=27997 exitstatus=0 id="admin/cluster-controllers/0"
distributor state=RUNNING mode=AUTO pid=27996 exitstatus=0 id="search/distributor/0"
logd state=RUNNING mode=AUTO pid=5751 exitstatus=0 id="hosts/r6-3/logd"
logserver state=RUNNING mode=AUTO pid=27994 exitstatus=0 id="admin/logserver"
searchnode state=RUNNING mode=AUTO pid=27995 exitstatus=0 id="search/search/cluster.search/0"
slobrok state=RUNNING mode=AUTO pid=28000 exitstatus=0 id="admin/slobrok.0"
```
To learn more about the processes and services, see [files and processes](files-processes-and-ports.html). Use [vespa-model-inspect host _hostname_](../../reference/operations/self-managed/tools.html#vespa-model-inspect) to list services running on a node.
##### Cluster startup
The config sentinel will not start services on a node unless it has connectivity to a minimum of other nodes, default 50%. Find an example of this feature in the [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA#start-the-admin-server) example application. Example configuration:
```xml
20
1
```
Example: `minOkPercent 10` means that services will be started only if more than or equal to 10% of nodes are up. If there are 11 nodes in the application, the first node started will not start its services - when the second node is started, services will be started on both.
`maxBadCount` is for connectivity checks where the other node is up, but we still do not have proper two-way connectivity. Normally, one-way connectivity means network configuration is broken and needs looking into, so this may be set low (1 or even 0 are the recommended values). If there are some temporary problems (in the example below non-responding DNS which leads to various issues at startup) the config sentinel will loop and retry, so the service startup will just be slightly delayed.
Example log:
```
[2021-06-15 14:33:25] EVENT : starting/1 name="sbin/vespa-config-sentinel -c hosts/le40808.ostk (pid 867)"
[2021-06-15 14:33:25] EVENT : started/1 name="config-sentinel"
[2021-06-15 14:33:25] CONFIG : Sentinel got 4 service elements [tenant(footest), application(bartest), instance(default)] for config generation 1001
[2021-06-15 14:33:25] CONFIG : Booting sentinel 'hosts/le40808.ostk' with [stateserver port 19098] and [rpc port 19097]
[2021-06-15 14:33:25] CONFIG : listening on port 19097
[2021-06-15 14:33:25] CONFIG : Sentinel got model info [version 7.420.21] for 35 hosts [config generation 1001]
[2021-06-15 14:33:25] CONFIG : connectivity.maxBadCount = 3
[2021-06-15 14:33:25] CONFIG : connectivity.minOkPercent = 40
[2021-06-15 14:33:28] INFO : Connectivity check details: 2086533.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le01287.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le23256.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le23267.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le23297.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le23312.ostk -> connect OK, but reverse check FAILED
[2021-06-15 14:33:28] INFO : Connectivity check details: le23317.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le23319.ostk -> connect OK, but reverse check FAILED
[2021-06-15 14:33:28] INFO : Connectivity check details: le30550.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le30553.ostk -> connect OK, but reverse check FAILED
[2021-06-15 14:33:28] INFO : Connectivity check details: le30556.ostk -> unreachable from me, but up
[2021-06-15 14:33:28] INFO : Connectivity check details: le30560.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le30567.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le40387.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le40389.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le40808.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le40817.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le40833.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le40834.ostk -> unreachable from me, but up
[2021-06-15 14:33:28] INFO : Connectivity check details: le40841.ostk -> connect OK, but reverse check FAILED
[2021-06-15 14:33:28] INFO : Connectivity check details: le40858.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le40860.ostk -> unreachable from me, but up
[2021-06-15 14:33:28] INFO : Connectivity check details: le40863.ostk -> connect OK, but reverse check FAILED
[2021-06-15 14:33:28] INFO : Connectivity check details: le40873.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le40892.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le40900.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le40905.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: le40914.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: sm02318.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: sm02324.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: sm02340.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: zt40672.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: zt40712.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: zt40728.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] INFO : Connectivity check details: zt41329.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:28] WARNING : 8 of 35 nodes up but with network connectivity problems (max is 3)
[2021-06-15 14:33:28] WARNING : Bad network connectivity (try 1)
[2021-06-15 14:33:30] WARNING : slow resolve time: 'le30556.ostk' -> '1234:5678:90:123::abcd' (5.00528 s)
[2021-06-15 14:33:30] WARNING : slow resolve time: 'le40834.ostk' -> '1234:5678:90:456::efab' (5.00527 s)
[2021-06-15 14:33:30] WARNING : slow resolve time: 'le40860.ostk' -> '1234:5678:90:789::cdef' (5.00459 s)
[2021-06-15 14:33:31] INFO : Connectivity check details: le23312.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:31] INFO : Connectivity check details: le23319.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:31] INFO : Connectivity check details: le30553.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:31] INFO : Connectivity check details: le30556.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:31] INFO : Connectivity check details: le40834.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:31] INFO : Connectivity check details: le40841.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:31] INFO : Connectivity check details: le40860.ostk -> connect OK, but reverse check FAILED
[2021-06-15 14:33:31] INFO : Connectivity check details: le40863.ostk -> OK: both ways connectivity verified
[2021-06-15 14:33:31] INFO : Enough connectivity checks OK, proceeding with service startup
[2021-06-15 14:33:31] EVENT : starting/1 name="searchnode"
...
```
Copyright © 2026 - [Cookie Preferences](#)
---
### Configuration Servers
[](/en/operations/self-managed/configuration-server.html.md "View as Markdown")
#### Configuration Servers
[](/en/operations/self-managed/configuration-server.html.md "View as Markdown")
Vespa Configuration Servers host the endpoint where application packages are deployed - and serves generated configuration to all services - see the [overview](../../learn/overview.html) and [application packages](../../basics/applications.html) for details. I.e., one cannot configure Vespa without config servers, and services cannot run without it.
It is useful to understand the [Vespa start sequence](config-sentinel.html). Refer to the sample applications [multinode](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode) and [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) for practical examples of multi-configserver configuration.
Vespa configuration is set up using one or more configuration servers (config servers). A config server uses [Apache ZooKeeper](https://zookeeper.apache.org/) as a distributed data storage for the configuration system. In addition, each node runs a config proxy to cache configuration data - find an overview at [services start](config-sentinel.html).
##### Status and config generation
Check the health of a running config server using (replace localhost with hostname):
```
$ curl http://localhost:19071/state/v1/health
```
Note that the config server is a service is itself, and runs with file-based configuration. The application packages deployed will not change the config server - the config server serves this configuration to all other Vespa nodes. This will hence always be config generation 0:
```
$ curl http://localhost:19071/state/v1/config
```
Details in [start-configserver](https://github.com/vespa-engine/vespa/blob/master/configserver/src/main/sh/start-configserver).
##### Redundancy
The config servers are defined in [VESPA\_CONFIGSERVERS](files-processes-and-ports.html#environment-variables), [services.xml](../../reference/applications/services/services.html) and [hosts.xml](/en/reference/applications/hosts.html):
```
$ VESPA_CONFIGSERVERS=myserver0.mydomain.com,myserver1.mydomain.com,myserver2.mydomain.com
```
```xml
```
```xml
admin0
admin1
admin2
```
[VESPA\_CONFIGSERVERS](files-processes-and-ports.html#environment-variables) must be set on all nodes. This is a comma- or whitespace-separated list with the hostname of all config servers, like _myhost1.mydomain.com,myhost2.mydomain.com,myhost3.mydomain.com_.
When there are multiple config servers, the [config proxy](config-proxy.html) will pick a config server randomly (to achieve load balancing between config servers). The config proxy is fault-tolerant and will switch to another config server (if there is more than one) if the one it is using becomes unavailable or there is an error in the configuration it receives.
For the system to tolerate _n_ failures, [ZooKeeper](#zookeeper) by design requires using _(2\*n)+1_ nodes. Consequently, only an odd numbers of nodes is useful, so you need minimum 3 nodes to have a fault-tolerant config system.
Even when using just one config server, the application will work if the server goes down (but deploying application changes will not work). Since the _config proxy_ runs on every node and caches configs, it will continue to serve config to the services on that node. However, restarting a node when config servers are unavailable means that services on the node will be unable to start since the cache will be destroyed when restarting the config proxy.
Refer to the [admin model reference](../../reference/applications/services/admin.html#configservers) for more details on _services.xml_.
##### Start sequence
To bootstrap a Vespa application instance, the high-level steps are:
- Start config servers
- Deploy config
- Start Vespa nodes
[multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) is a great guide on how to start a multinode Vespa application instance - try this first. Detailed steps for config server startup:
1. Set [VESPA\_CONFIGSERVERS](files-processes-and-ports.html#environment-variables) on all nodes, using fully qualified hostnames and the same value on all nodes, including the config servers.
2. Start the config server on the nodes configured in _services/hosts.xml_. Make sure the startup is successful by inspecting [/state/v1/health](../../reference/api/state-v1.html#state-v1-health), default on port 19071:
```
$ curl http://localhost:19071/state/v1/health
```
```json
{
"time" : 1651147368066,
"status" : {
"code" : "up"
},
"metrics" : {
"snapshot" : {
"from" : 1.651147308063E9,
"to" : 1.651147367996E9
}
}
}
```
If there is no response on the health API, two things can have happened:
- The config server process did not start - inspect logs using `vespa-logfmt`, or check _$VESPA\_HOME/logs/vespa/vespa.log_, normally _/opt/vespa/logs/vespa/vespa.log_.
- The config server process started, and is waiting for [Zookeeper quorum](#zookeeper):
```
$ vespa-logfmt -S configserver
```
```
configserver Container.com.yahoo.vespa.zookeeper.ZooKeeperRunner Starting ZooKeeper server with /opt/vespa/var/zookeeper/conf/zookeeper.cfg. Trying to establish ZooKeeper quorum (members: [node0.vespanet, node1.vespanet, node2.vespanet], attempt 1)configserver Container.com.yahoo.container.handler.threadpool.ContainerThreadpoolImpl Threadpool 'default-pool': min=12, max=600, queue=0
configserver Container.com.yahoo.vespa.config.server.tenant.TenantRepository Adding tenant 'default', created 2022-04-28T13:02:24.182Z. Bootstrapping in PT0.175576S
configserver Container.com.yahoo.vespa.config.server.rpc.RpcServer Rpc server will listen on port 19070
configserver Container.com.yahoo.container.jdisc.state.StateMonitor Changing health status code from 'initializing' to 'up'
configserver Container.com.yahoo.jdisc.http.server.jetty.Janitor Creating janitor executor with 2 threads
configserver Container.com.yahoo.jdisc.http.server.jetty.JettyHttpServer Threadpool size: min=22, max=22
configserver Container.org.eclipse.jetty.server.Server jetty-9.4.46.v20220331; built: 2022-03-31T16:38:08.030Z; git: bc17a0369a11ecf40bb92c839b9ef0a8ac50ea18; jvm 11.0.14.1+1-
configserver Container.org.eclipse.jetty.server.handler.ContextHandler Started o.e.j.s.ServletContextHandler@341c0dfc{19071,/,null,AVAILABLE}
configserver Container.org.eclipse.jetty.server.AbstractConnector Started configserver@3cd6d147{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:19071}
configserver Container.org.eclipse.jetty.server.Server Started @21955ms
configserver Container.com.yahoo.container.jdisc.ConfiguredApplication Switching to the latest deployed set of configurations and components.Application config generation: 0
```
It will hang until quorum is reached, and the second highlighted log line is emitted. Root causes for missing quorum can be:
- No connectivity between the config servers. Zookeeper logs the members like `(members: [node0.vespanet, node1.vespanet, node2.vespanet], attempt 1)`. Verify that the nodes running config server can reach each other on port 2181.
- No connectivity can be wrong network config. [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) uses a docker network, make sure there are no underscores in the hostnames.
3. Once all config servers return `up` on _state/v1/health_, an application package can be deployed. This means, if deploy fails, it is always a good idea to verify the config server health first - if config servers are up, and deploy fails, it is most likely an issue with the application package - if so, refer to [application packages](../../basics/applications.html).
4. A successful deployment logs the following, for the _prepare_ and _activate_ steps:
```
Container.com.yahoo.vespa.config.server.ApplicationRepository Session 2 prepared successfully.
Container.com.yahoo.vespa.config.server.deploy.Deployment Session 2 activated successfully using no host provisioner. Config generation 2. File references: [file '9cfc8dc57f415c72']
Container.com.yahoo.vespa.config.server.session.SessionRepository Session activated: 2
```
5. Start the Vespa nodes. Technically, they can be started at any time. When troubleshooting, it is easier to make sure the config servers are started successfully, and deployment was successful - before starting any other nodes. Refer to the [Vespa start sequence](config-sentinel.html) and [Vespa start / stop / restart](admin-procedures.html#vespa-start-stop-restart).
Make sure to look for logs on all config servers when debugging.
##### Scaling up
Add a config server node for increased fault tolerance or when replacing a node. Read up on [ZooKeeper configuration](#zookeeper-configuration) before continuing. Although it is _possible_ to add more than one config server at a time, doing it one by one is recommended, to keep the ZooKeeper quorum intact.
Due to the ZooKeeper majority vote, use one or three config servers.
1. Install _vespa_ on new config server node.
2. Append the config server node's hostname to VESPA\_CONFIGSERVERS on all nodes, then (re)start all config servers in sequence to update the ZooKeeper config. By appending, the current config server nodes keep their current ZooKeeper index. Restart the existing config server(s) first. Config server will log which servers are configured when starting up to vespa log.
3. Update _services.xml_ and _hosts.xml_ with the new set of config servers, then _vespa prepare_ and _vespa activate_.
4. Restart other nodes one by one to start using the new config servers. This will let the vespa nodes use the updated set of config servers.
The config servers will automatically redistribute the application data to new nodes.
##### Scaling down
This is the inverse of scaling up, and the procedure is the same. Remove config servers from the end of _VESPA\_CONFIGSERVERS_, and here one can remove two nodes in one go, if going from three to one.
##### Replacing nodes
- Make sure to replace only one node at a time.
- If you have only one config server you need to first scale up with a new node, then scale down by removing the old node.
- If you have 3 or more you can replace one of the old nodes in VESPA\_CONFIGSERVERS with the new one instead of adding one, otherwise same procedure as in [Scaling up](#scaling-up). Repeat for each node you want to replace.
##### Tools
Tools to access config:
- [vespa-get-config](../../reference/operations/self-managed/tools.html#vespa-get-config)
- [vespa-configproxy-cmd](../../reference/operations/self-managed/tools.html#vespa-configproxy-cmd)
- [Config API](../../reference/api/config-v2.html)
##### ZooKeeper
[ZooKeeper](https://zookeeper.apache.org/) handles data consistency across multiple config servers. The config server Java application runs a ZooKeeper server, embedded with an RPC frontend that the other nodes use. ZooKeeper stores data internally in _nodes_ that can have _sub-nodes_, similar to a file system.
At [vespa prepare](../../reference/clients/vespa-cli/vespa_prepare), the application's files, along with global configurations, are stored in ZooKeeper. The application data is stored under _/config/v2/tenants/default/sessions/[sessionid]/userapp_. At [vespa activate](../../reference/clients/vespa-cli/vespa_activate), the newest application is activated _live_ by writing the session id into _/config/v2/tenants/default/applications/default:default:default_. It is at that point the other nodes get configured.
Use _vespa-zkcli_ to inspect state, replace with actual session id:
```
$ vespa-zkcli ls /config/v2/tenants/default/sessions/sessionid/userapp
$ vespa-zkcli get /config/v2/tenants/default/sessions/sessionid/userapp/services.xml
```
The ZooKeeper server logs to _$VESPA\_HOME/logs/vespa/zookeeper.configserver.0.log (files are rotated with sequence number)_
###### ZooKeeper configuration
The members of the ZooKeeper cluster is generated based on the contents of [VESPA\_CONFIGSERVERS](files-processes-and-ports.html#environment-variables). _$VESPA\_HOME/var/zookeeper/conf/zookeeper.cfg_ is written when (re)starting the config server. Hence, config server(s) must all be restarted when `VESPA_CONFIGSERVERS` changes.
The order of the nodes is used to create indexes in _zookeeper.cfg_, do not change node order.
###### ZooKeeper recovery
If the config server(s) should experience data corruption, for instance a hardware failure, use the following recovery procedure. One example of such a scenario is if _$VESPA\_HOME/logs/vespa/zookeeper.configserver.0.log_ says _java.io.IOException: Negative seek offset at java.io.RandomAccessFile.seek(Native Method)_, which indicates ZooKeeper has not been able to recover after a full disk. There is no need to restart Vespa on other nodes during the procedure:
1. [vespa-stop-configserver](../../reference/operations/self-managed/tools.html#vespa-stop-configserver)
2. [vespa-configserver-remove-state](../../reference/operations/self-managed/tools.html#vespa-configserver-remove-state)
3. [vespa-start-configserver](../../reference/operations/self-managed/tools.html#vespa-start-configserver)
4. [vespa](../../clients/vespa-cli.html#deployment) prepare \
5. [vespa](../../clients/vespa-cli.html#deployment) activate
This procedure completely cleans out ZooKeeper's internal data snapshots and deploys from scratch.
Note that by default the [cluster controller](../../content/content-nodes.html#cluster-controller) that maintains the state of the content cluster will use the shared same ZooKeeper instance, so the content cluster state is also reset when removing state. Manually set state will be lost (e.g. a node with user state _down_). It is possible to run cluster-controllers in standalone zookeeper mode - see [standalone-zookeeper](../../reference/applications/services/admin.html#cluster-controllers).
###### ZooKeeper barrier timeout
If the config servers are heavily loaded, or the applications being deployed are big, the internals of the server may time out when synchronizing with the other servers during deploy. To work around, increase the timeout by setting: [VESPA\_CONFIGSERVER\_ZOOKEEPER\_BARRIER\_TIMEOUT](files-processes-and-ports.html#environment-variables) to 600 (seconds) or higher, and restart the config servers.
##### Configuration
To access config from a node not running the config system (e.g. doing feeding via the Document API), use the environment variable [VESPA\_CONFIG\_SOURCES](files-processes-and-ports.html#environment-variables):
```
$ export VESPA_CONFIG_SOURCES="myadmin0.mydomain.com:19071,myadmin1.mydomain.com:19071"
```
Alternatively, for Java programs, use the system property _configsources_ and set it programmatically or on the command line with the _-D_ option to Java. The syntax for the value is the same as for _VESPA\_CONFIG\_SOURCES_.
###### System requirements
The minimum heap size for the JVM it runs under is 128 Mb and max heap size is 2 GB (which can be changed with a [setting](../../performance/container-tuning.html#config-server-and-config-proxy)). It writes a transaction log that is regularly purged of old items, so little disk space is required. Note that running on a server that has a lot of disk I/O will adversely affect performance and is not recommended.
###### Ports
The config server RPC port can be changed by setting [VESPA\_CONFIGSERVER\_RPC\_PORT](files-processes-and-ports.html#environment-variables) on all nodes in the system.
Changing HTTP port requires changing the port in _$VESPA\_HOME/conf/configserver-app/services.xml_:
```xml
```
When deploying, use the _-p_ option, if port is changed from the default.
##### Troubleshooting
| Problem | Description |
| --- | --- |
| Health checks |
Verify that a config server is up and running using [/state/v1/health](../../reference/api/state-v1.html#state-v1-health), see [start sequence](#start-sequence). Status code is `up` if the server is up and has finished bootstrapping.
Alternatively, use [http://localhost:19071/status.html](http://localhost:19071/status.html) which will return response code 200 if server is up and has finished bootstrapping.
Metrics are found at [/state/v1/metrics](../../reference/api/state-v1.html#state-v1-metrics). Use [vespa-model-inspect](../../reference/operations/self-managed/tools.html#vespa-model-inspect) to find host and port number, port is 19071 by default.
|
| Consistency |
When having more than one config server, consistency between the servers is crucial. [http://localhost:19071/status](http://localhost:19071/status) can be used to check that settings for config servers are the same for all servers.
[vespa-config-status](../../reference/operations/self-managed/tools.html#vespa-config-status) can be used to check config on nodes.
[http://localhost:19071/application/v2/tenant/default/application/default](http://localhost:19071/application/v2/tenant/default/application/default) displays active config generation and should be the same on all servers, and the same as in response from running [vespa deploy](../../clients/vespa-cli.html#deployment)
|
| Bad Node |
If running with more than one config server and one of these goes down or has hardware failure, the cluster will still work and serve config as usual (clients will switch to use one of the good servers). It is not necessary to remove a bad server from the configuration.
Deploying applications will take longer, as [vespa deploy](../../clients/vespa-cli.html#deployment) will not be able to complete a deployment on all servers when one of them is down. If this is troublesome, lower the [barrier timeout](#zookeeper-barrier-timeout) - (default value is 120 seconds).
Note also that if you have not configured [cluster controllers](../../reference/applications/services/admin.html#cluster-controller) explicitly, these will run on the config server nodes and the operation of these might be affected. This is another reason for not trying to manually remove a bad node from the config server setup.
|
| Stuck filedistribution |
The config system distributes binary files (such as jar bundle files) using [file-distribution](../../applications/deployment.html#file-distribution) - use [vespa-status-filedistribution](../../reference/operations/self-managed/tools.html#vespa-status-filedistribution) to see detailed status if it gets stuck.
|
| Memory |
Insufficient memory on the host / in the container running the config server will cause startup or deploy / configuration problems - see [Docker containers](docker-containers.html).
|
| ZooKeeper |
The following can be caused by a full disk on the config server, or clocks out of sync:
```
at com.yahoo.vespa.zookeeper.ZooKeeperRunner.startServer(ZooKeeperRunner.java:92)
Caused by: java.io.IOException: The accepted epoch, 10 is less than the current epoch, 48
```
Users have reported that "Copying the currentEpoch to acceptedEpoch fixed the problem".
|
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Configuration Servers](#page-title)
- [Status and config generation](#status-and-config-generation)
- [Redundancy](#redundancy)
- [Start sequence](#start-sequence)
- [Scaling up](#scaling-up)
- [Scaling down](#scaling-down)
- [Replacing nodes](#replacing-nodes)
- [Tools](#tools)
- [ZooKeeper](#zookeeper)
- [ZooKeeper configuration](#zookeeper-configuration)
- [ZooKeeper recovery](#zookeeper-recovery)
- [ZooKeeper barrier timeout](#zookeeper-barrier-timeout)
- [Configuration](#configuration)
- [System requirements](#system-requirements)
- [Ports](#ports)
- [Troubleshooting](#troubleshooting)
---
### Container
[](/en/operations/self-managed/container.html.md "View as Markdown")
#### Container
[](/en/operations/self-managed/container.html.md "View as Markdown")
This is the Container service operational guide.

Note that "container" is an overloaded concept in Vespa - in this guide it refers to service instance nodes in blue.
Refer to [container metrics](../metrics.html#container-metrics).
##### Endpoints
Container service(s) hosts the query and feed endpoints - examples:
- [album-recommendation](https://github.com/vespa-engine/sample-apps/blob/master/album-recommendation/app/services.xml) configures \_both\_ query and feed in the same container cluster (i.e. service):
```xml
```
- [multinode-HA](https://github.com/vespa-engine/sample-apps/blob/master/examples/operations/multinode-HA/services.xml) configures query and feed in separate container clusters (i.e. services):
```xml
```
Observe that `` and `` are located in separate clusters in the second example, and endpoints are therefore different.
**Important:** The first thing to validate when troubleshooting query errors is to make sure that the endpoint is correct, i.e. that query requests hit the correct nodes. A query will be written to the [access log](../access-logging.html) on one of the nodes in the container cluster
##### Inspecting Vespa Java Services using JConsole
Determine the state of each running Java Vespa service using JConsole. JConsole is distributed along with the Java developer kit. Start JConsole:
```
$ jconsole :
```
where the host and port determine which service to attach to. For security purposes the JConsole tool can not directly attach to Vespa services from external machines.
###### Connecting to a Vespa instance
To attach a JConsole to a Vespa service running on another host, create a tunnel from the JConsole host to the Vespa service host. This can for example be done by setting up two SSH tunnels as follows:
```
$ ssh -N -L:localhost: &
$ ssh -N -L:localhost: &
```
where port1 and port2 are determined by the type of service (see below). A JConsole can then be attached to the service as follows:
```
$ jconsole localhost:
```
Port numbers:
| Service | Port 1 | Port 2 |
| --- | --- | --- |
| QRS | 19015 | 19016 |
| Docproc | 19123 | 19124 |
Updated port information can be found by running:
```
$[vespa-model-inspect](../../reference/operations/self-managed/tools.html#vespa-model-inspect)service
```
where the resulting RMIREGISTRY and JMX lines determine port1 and port2, respectively.
###### Examining thread states
The state of each container is available in JConsole by pressing the Threads tab and selecting the thread of interest in the threads list. Threads of interest includes _search_, _connector_, _closer_, _transport_ and _acceptor_ (the latter four are used for backend communications).
Copyright © 2026 - [Cookie Preferences](#)
---
### Content node recovery
[](/en/operations/self-managed/content-node-recovery.html.md "View as Markdown")
#### Content node recovery
[](/en/operations/self-managed/content-node-recovery.html.md "View as Markdown")
In exceptional cases, one or more content nodes may end up with corrupted data causing it to fail to restart. Possible reasons are
- the application configuring a higher memory or disk limit such that the node is allowed to accept more data than it can manage,
- hardware failure, or
- a bug in Vespa.
Normally a corrupted node can just be wiped of all data or removed from the cluster, but when this happens simultaneously to multiple nodes, or redundancy 1 is used, it may be necessary to recover the node(s) to avoid data loss. This documents explains the procedure.
##### Recovery steps
On each of the nodes needing recovery:
1. [Stop services](admin-procedures.html#vespa-start-stop-restart) on the node if running.
2. Repair the node:
- If the node cannot start due to needing more memory than available: Increase the memory available to the node, or if not possible stop all non-essential processes on the node using `vespa-sentinel-cmd list` and `vespa-sentinel-cmd stop [name]`, and (if necessary) start only the content node process using `vespa-sentinel-cmd start searchnode`. When the node is successfully started, issue delete operations or increase the cluster size to reduce the amount of data on the node if necessary.
- If the node cannot start due to needing more disk than available: Increase the disk available to the node, or if not possible delete non-essential data such as logs and cached packages. When the node is successfully started, issue delete operations or increase the cluster size to reduce the amount of data on the node if necessary.
- If the node cannot start for any other reason, repair the data manually as needed. This procedure will depend on the specific nature of the data corruption.
3. [Start services](admin-procedures.html#vespa-start-stop-restart) on the node.
4. Verify that the node is fully up before doing the next node - metrics/interfaces to be used to evaluate if the next node can be stopped:
- Check if a node is up using [/state/v1/health](../../reference/api/state-v1.html#state-v1-health).
- Check the `vds.idealstate.merge_bucket.pending.average` metric on content nodes. When 0, all buckets are in sync - see [example](../metrics.html).
Copyright © 2026 - [Cookie Preferences](#)
---
### CPU Support
[](/en/operations/self-managed/cpu-support.html.md "View as Markdown")
#### CPU Support
[](/en/operations/self-managed/cpu-support.html.md "View as Markdown")
For maximum performance, the current version of Vespa for x86\_64 is compiled only for [Haswell (2013)](https://en.wikipedia.org/wiki/Haswell_(microarchitecture)) or later CPUs. If trying to run on an older CPU, you will likely see error messages like the following:
```
Problem running program /opt/vespa/bin/vespa-runserver => died with signal: illegal instruction (you probably have an older CPU than required)
```
or in older versions of Vespa, something like
```
/usr/local/bin/start-container.sh: line 67: 10 Illegal instruction /opt/vespa/bin/vespa-start-configserver
```
If you would like to run Vespa on an older CPU, we provide a [generic x86 container image](https://hub.docker.com/r/vespaengine/vespa-generic-intel-x86_64/). This image is slower, receives less testing than the regular image, and is less frequently updated.
**To start a Vespa Docker container using this image:**
```
$ docker run --detach --name vespa --hostname vespa-container \
--publish 8080:8080 --publish 19071:19071 \
vespaengine/vespa-generic-intel-x86_64
```
Copyright © 2026 - [Cookie Preferences](#)
---
### Docker containers
[](/en/operations/self-managed/docker-containers.html.md "View as Markdown")
#### Docker containers
[](/en/operations/self-managed/docker-containers.html.md "View as Markdown")
This document describes tuning and adaptions for running Vespa Docker containers, for developer use on laptop, and in production.
##### Mounting persistent volumes
The [quick start](../../basics/deploy-an-application-local.html) and [AWS ECS multinode](multinode-systems.html#aws-ecs) guides show how to run Vespa in Docker containers. In these examples, all the data is stored inside the container - the data is lost if the container is deleted. When running Vespa inside Docker containers in production, volume mappings to the parent host should be added to persist data and logs.
- /opt/vespa/var
- /opt/vespa/logs
```
$ mkdir -p /tmp/vespa/var; export VESPA_VAR_STORAGE=/tmp/vespa/var
$ mkdir -p /tmp/vespa/logs; export VESPA_LOG_STORAGE=/tmp/vespa/logs
$ docker run --detach --name vespa --hostname vespa-container \
--volume $VESPA_VAR_STORAGE:/opt/vespa/var \
--volume $VESPA_LOG_STORAGE:/opt/vespa/logs \
--publish 8080:8080 \
vespaengine/vespa
```
##### Start Vespa container with Vespa user
You can start the container directly as the _vespa_ user. The _vespa_ user and group within the container are configured with user id _1000_ and group id _1000_. The vespa user and group must be the owner of the _/opt/vespa/var_ and _/opt/vespa/logs_ volumes that are mounted in the container for Vespa to start. This is required for Vespa to create the required directories and files within those directories.
The start script will check that the correct owner uid and gid are set and fail if the wrong user or group is set as the owner.
When using an isolated user namespace for the Vespa container, you must set the uid and gid of the directories on the host to the subordinate uid and gid, depending on your mapping. See the [Docker documentation](https://docs.docker.com/engine/security/userns-remap/) for more details.
```
$ mkdir -p /tmp/vespa/var; export VESPA_VAR_STORAGE=/tmp/vespa/var
$ mkdir -p /tmp/vespa/logs; export VESPA_LOG_STORAGE=/tmp/vespa/logs
$ sudo chown -R 1000:1000 $VESPA_VAR_STORAGE $VESPA_LOG_STORAGE
$ docker run --detach --name vespa --user vespa:vespa --hostname vespa-container \
--volume $VESPA_VAR_STORAGE:/opt/vespa/var \
--volume $VESPA_LOG_STORAGE:/opt/vespa/logs \
--publish 8080:8080 \
vespaengine/vespa
```
##### System limits
When Vespa starts inside Docker containers, the startup scripts will set [system limits](files-processes-and-ports.html#vespa-system-limits). Make sure that the environment starting the Docker engine is set up in such a way that these limits can be set inside the containers.
For a CentOS/RHEL base host, Docker is usually started by [systemd](https://www.freedesktop.org/software/systemd/man/systemd.exec.html). In this case, `LimitNOFILE`, `LimitNPROC` and `LimitCORE` should be set to meet the minimum requirements in [system limits](files-processes-and-ports.html#vespa-system-limits).
In general, when using Docker or Podman to run Vespa, the `--ulimit` option should be used to set limits according to [system limits](files-processes-and-ports.html#vespa-system-limits). The `--pids-limit` should be set to unlimited (`-1` for Docker and `0` for Podman).
##### Transparent Huge Pages
Vespa performance improves significantly by enabling [Transparent Huge Pages (THP)](https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html), especially for memory-intensive applications with large dense tensors with concurrent query and write workloads.
One application improved query p99 latency from 950 ms to 150 ms during concurrent query and write by enabling THP. Using THP is even more important when running in virtualized environments like AWS and GCP due to nested page tables.
When running Vespa using the container image, _THP_ settings must be set on the base host OS (Linux). The recommended settings are:
```
$ echo 1 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag
$ echo always > /sys/kernel/mm/transparent_hugepage/enabled
$ echo never > /sys/kernel/mm/transparent_hugepage/defrag
```
To verify that the setting is active, one should see that _AnonHugePages_ is non-zero, In this case, 75 GB has been allocated using AnonHugePages.
```
$ cat /proc/meminfo |grep AnonHuge
AnonHugePages: 75986944 kB
```
Note that the Vespa container needs to be restarted after modifying the base host OS settings to make the changes effective. Vespa uses `MADV_HUGEPAGE` for memory allocations done by the [content node process (proton)](/en/content/proton.html).
##### Controlling which services to start
The Docker image _vespaengine/vespa_'s [start script](https://github.com/vespa-engine/docker-image/blob/master/include/start-container.sh) takes a parameter that controls which services are started inside the container.
Starting a _configserver_ container:
```
$ docker run \
--env VESPA_CONFIGSERVERS= \
vespaengine/vespaconfigserver
```
Starting a _services_ container (configserver will not be started):
```
$ docker run \
--env VESPA_CONFIGSERVERS= \
vespaengine/vespaservices
```
Starting a container with _both configserver and services_:
```
$ docker run \
--env VESPA_CONFIGSERVERS= \
vespaengine/vespaconfigserver,services
```
This is required in the case where the configserver container should run other services like an adminserver or logserver (see [services.html](/en/reference/applications/services/services.html))
If the [VESPA\_CONFIGSERVERS](files-processes-and-ports.html#environment-variables) environment variable is not specified, it will be set to the container hostname, also see [node setup](node-setup.html#hostname).
Use the [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) sample application as a blueprint for how to set up config servers and services.
##### Graceful stop
Stopping a running _vespaengine/vespa_ container triggers a graceful shutdown, which saves time when starting the container again (i.e., data structures are flushed). If the container is shut down forcefully, the content nodes might need to restore the state from the transaction log, which might be time-consuming. There is no chance of data loss or data corruption as the data is always written and synced to persistent storage.
The default timeout for the Docker daemon to wait for the shutdown might be too low for larger number of documents per node. Below stop will wait at least 120 seconds before terminating the running container forcefully, if the stop is successfully performed before the timeout has passed, the command takes less than the timeout:
```
$ docker stop name -t 120
```
It is also possible to configure the default Docker daemon timeout, see [--shutdown-timeout](https://docs.docker.com/reference/cli/dockerd/).
A clean content node shutdown looks like:
```
[2025-05-02 10:07:52.052] EVENT searchnode proton.node.server stopping/1 name="storagenode" why="Stopped"
[2025-05-02 10:07:52.056] EVENT searchnode proton stopping/1 name="servicelayer" why="clean shutdown"
[2025-05-02 10:07:52.056] INFO searchnode proton.proton.server.rtchooks shutting down monitoring interface
[2025-05-02 10:07:52.058] INFO searchnode proton.searchlib.docstore.logdatastore Flushing. Disk bloat is now at 0 of 8832 at 0.00 percent
[2025-05-02 10:07:52.059] INFO searchnode proton.searchlib.docstore.logdatastore Flushing. Disk bloat is now at 0 of 8832 at 0.00 percent
[2025-05-02 10:07:52.060] INFO searchnode proton.searchlib.docstore.logdatastore Flushing. Disk bloat is now at 0 of 8840 at 0.00 percent
[2025-05-02 10:07:52.066] INFO searchnode proton.transactionlog.server Stopping TLS
[2025-05-02 10:07:52.066] INFO searchnode proton.transactionlog.server TLS Stopped
[2025-05-02 10:07:52.071] EVENT searchnode proton stopping/1 name="proton" why="clean shutdown"
[2025-05-02 10:07:52.078] EVENT config-sentinel sentinel.sentinel.service stopped/1 name="searchnode" pid=354 exitcode=0
```
##### Memory
The [sample applications](https://github.com/vespa-engine/sample-apps) and [local application deployment guide](../../basics/deploy-an-application-local.html) indicates the minimum memory requirements for the Docker containers.
**Note:** Too little memory is a very common problem when testing Vespa in Docker containers. Use the below to troubleshoot before making a support request, and also see the [FAQ](../../learn/faq).
As a rule of thumb, a single-node Vespa application requires a minimum of 4 GB for the Docker container. Using `docker stats` can be useful to track memory usage:
```
$ docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
589bf5801b22 node0 213.25% 697.3MiB / 3.84GiB 17.73% 14.2kB / 11.5kB 617MB / 976MB 253
e108dde84679 node1 213.52% 492.7MiB / 3.84GiB 12.53% 15.7kB / 12.7kB 74.3MB / 924MB 252
be43aacd0bbb node2 191.22% 497.8MiB / 3.84GiB 12.66% 19.6kB / 21.6kB 64MB / 949MB 261
```
It is not necessarily easy to verify that Vespa has started all services successfully. Symptoms of errors due to insufficient memory vary, depending on where it fails. Example: Inspect restart logs in a container named _vespa_, running the [quickstart](../../basics/deploy-an-application-local.html) with only 2G:
```
$ docker exec -it vespa sh -c "/opt/vespa/bin/vespa-logfmt -S config-sentinel -c sentinel.sentinel.service"
INFO : config-sentinel sentinel.sentinel.service container: incremented restart penalty to 2.000 seconds
INFO : config-sentinel sentinel.sentinel.service container: incremented restart penalty to 6.000 seconds
INFO : config-sentinel sentinel.sentinel.service container: incremented restart penalty to 14.000 seconds
INFO : config-sentinel sentinel.sentinel.service container: incremented restart penalty to 30.000 seconds
INFO : config-sentinel sentinel.sentinel.service container: will delay start by 25.173 seconds
INFO : config-sentinel sentinel.sentinel.service container: incremented restart penalty to 62.000 seconds
INFO : config-sentinel sentinel.sentinel.service container: incremented restart penalty to 126.000 seconds
INFO : config-sentinel sentinel.sentinel.service container: will delay start by 119.515 seconds
INFO : config-sentinel sentinel.sentinel.service container: incremented restart penalty to 254.000 seconds
INFO : config-sentinel sentinel.sentinel.service container: incremented restart penalty to 510.000 seconds
INFO : config-sentinel sentinel.sentinel.service container: will delay start by 501.026 seconds
INFO : config-sentinel sentinel.sentinel.service container: incremented restart penalty to 1022.000 seconds
INFO : config-sentinel sentinel.sentinel.service container: incremented restart penalty to 1800.000 seconds
INFO : config-sentinel sentinel.sentinel.service container: will delay start by 1793.142 seconds
```
Observe that the _container_ service restarts in a loop, with increasing pause.
A common problem is [config servers](configuration-server.html) not starting or running properly due to a lack of memory. This manifests itself as nothing listening on 19071, or deployment failures.
Some guides/sample applications have specific configurations to minimize resource usage. Example from [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA):
```
$ docker run --detach --name node0 --hostname node0.vespanet \
-e VESPA_CONFIGSERVERS=node0.vespanet,node1.vespanet,node2.vespanet \
-eVESPA\_CONFIGSERVER\_JVMARGS="-Xms32M -Xmx128M"\
-eVESPA\_CONFIGPROXY\_JVMARGS="-Xms32M -Xmx32M"\
--network vespanet \
--publish 19071:19071 --publish 19100:19100 --publish 19050:19050 --publish 20092:19092 \
vespaengine/vespa
```
Here [VESPA\_CONFIGSERVER\_JVMARGS](files-processes-and-ports.html#environment-variables) and [VESPA\_CONFIGPROXY\_JVMARGS](files-processes-and-ports.html#environment-variables) are tweaked to the minimum for a functional test only.
**Important:** For production use, do not reduce memory settings in `VESPA_CONFIGSERVER_JVMARGS` and `VESPA_CONFIGPROXY_JVMARGS` unless you know what you are doing - the Vespa defaults are set for regular production use, and rarely need changing.
Container memory setting are done in _services.xml_, example from [multinode-HA](https://github.com/vespa-engine/sample-apps/blob/master/examples/operations/multinode-HA/services.xml):
```xml
```
Make sure that the settings match the Docker container Vespa is running in.
Also see [node memory settings](node-setup.html#memory-settings) for more settings.
##### Network
Vespa processes communicate over both fixed and ephemeral ports - in general, all ports must be accessible. See [example ephemeral use](../../writing/visiting.html#handshake-failed).
Find an example application using a Docker network in [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA).
##### Resource usage
Note that CPU usage will not be zero even if there are zero documents and zero queries. Starting the _vespaengine/vespa_ container image means starting the [configuration server](configuration-server.html) and the [configuration sentinel](config-sentinel.html). When deploying an application, the sentinel starts the configured service processes, and they all listen to work to do, changes in the config, and so forth.
Therefore, an "idle" container instance consumes CPU and memory.
##### Troubleshooting
The Vespa documentation examples use `docker`. The Vespa Team has good experience with using `podman`, too, in the examples just change from `docker` to `podman`. We recommend using Podman v5, see the [release notes](https://github.com/containers/podman/blob/main/RELEASE_NOTES.md). [emulating-docker-cli-with-podman](https://podman-desktop.io/docs/migrating-from-docker/emulating-docker-cli-with-podman) is a useful resource.
Many startup failures are caused by a failed Vespa Container start due to configuration or download errors. Use `docker logs vespa` to show the log (this example assumes a Docker container named `vespa`, use `docker ps` to list containers).
###### Docker image
Make sure to use a recent Vespa release (check [releases](https://factory.vespa.ai/releases)) and validate the downloaded image:
```
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/vespaengine/vespa latest 8cfb0da22c01 35 hours ago 1.2 GB
```
###### Model download failures
If the application package depends on downloaded models, look for `RuntimeException: Not able to create config builder for payload` - [details](../../applications/components.html#component-load).
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Docker containers](#page-title)
- [Mounting persistent volumes](#mounting-persistent-volumes)
- [Start Vespa container with Vespa user](#start-vespa-container-with-vespa-user)
- [System limits](#system-limits)
- [Transparent Huge Pages](#transparent-huge-pages)
- [Controlling which services to start](#controlling-which-services-to-start)
- [Graceful stop](#graceful-stop)
- [Memory](#memory)
- [Network](#network)
- [Resource usage](#resource-usage)
- [Troubleshooting](#troubleshooting)
- [Docker image](#docker-image)
- [Model download failures](#model-download-failures)
---
### Files, Processes, Ports, Environment
[](/en/operations/self-managed/files-processes-and-ports.html.md "View as Markdown")
#### Files, Processes, Ports, Environment
[](/en/operations/self-managed/files-processes-and-ports.html.md "View as Markdown")
This is a reference of directories used in a Vespa installation, processes that run on the Vespa nodes and ports / environment variables used. Also see [log files](../../reference/operations/log-files.html).
##### Directories
| Directory | Description |
| --- | --- |
| $VESPA\_HOME/bin/ | Command line utilities and scripts |
| $VESPA\_HOME/libexec/vespa/ | Command line utilities and scripts |
| $VESPA\_HOME/sbin/ | Server programs, daemons, etc |
| $VESPA\_HOME/lib64/ | Dynamically linked libraries, typically third-party libraries |
| $VESPA\_HOME/lib/jars/ | Java archives |
| $VESPA\_HOME/logs/vespa/ | Log files |
| $VESPA\_HOME/var/db/vespa/config\_server/serverdb/ | Config server database and user applications |
| $VESPA\_HOME/share/vespa/ | A directory with config definitions and XML schemas for application package validation |
| $VESPA\_HOME/conf/vespa | Various config files used by Vespa or libraries Vespa depend on |
##### Processes and ports
The following is an overview of which ports and port ranges are used by the different services in a Vespa system. Note that for services capable of running multiple instances on the same node, all instances will run within the listed port range.
Processes are run as user `vespa`.
Many services are allocated ports dynamically. So even though the allocation is deterministic, i.e. the same system will get the same ports on subsequent startups, a particular service instance may get different ports when the overall system setup is changed through [services.xml](../../reference/applications/services/services.html). Use [vespa-model-inspect](../../reference/operations/self-managed/tools.html#vespa-model-inspect) to see port allocations.
- The number of ports used in a range depends on number of instances that are running
- Not all ports within a range are used, but they are assigned each service to support future extensions
- The range from 19100 is used for internal communication ports, i.e. ports that are not necessary to use from an external API
- See [Configuring Http Servers and Filters](../../applications/http-servers-and-filters.html) for how to configure Container ports and [services.xml](../../reference/applications/services/services.html) for how to configure other ports
| Process | Host | Port/range | ps | Function |
| --- | --- | --- | --- | --- |
| [Config server](configuration-server.html) | Config server nodes | 19070-19071 | java (...) -jar $VESPA\_HOME/lib/jars/standalone-container-jar-with-dependencies.jar | Vespa Configuration server |
| 2181-2183 | | Embedded Zookeeper cluster ports, see [zookeeper-server.def](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/zookeeper-server.def) |
| [Config sentinel](config-sentinel.html) | All nodes | 19098 | $VESPA\_HOME/sbin/vespa-config-sentinel | Sentinel that starts and stops vespa services and makes sure they are running unless they are manually stopped |
| [Config proxy](config-proxy.html) | All nodes | 19090 | java (…) com.yahoo.vespa.config.proxy.ProxyServer | Communication liaison between Vespa processes and config server. Caches config in memory |
| [Slobrok](slobrok.html) | Admin nodes | 19099 for RPC port, HTTP port dynamically allocated in the 19100-19899 range | $VESPA\_HOME/sbin/vespa-slobrok | Service location object broker |
| [logd](../../reference/operations/log-files.html#logd) | All nodes | 19089 | $VESPA\_HOME/sbin/vespa-logd | Reads local log files and sends them to log server |
| [Log server](../../reference/operations/log-files.html#log-server) | Log server node | 19080 | java (...) -jar lib/jars/logserver-jar-with-dependencies.jar | Vespa Log server |
| [Metrics proxy](monitoring.html#metrics-proxy) | All nodes | 19092-19095 | java (...) -jar $VESPA\_HOME/lib/jars/container-disc-with-dependencies.jar | Provides a single access point for metrics from all services on a Vespa node |
| [Distributor](../../content/content-nodes.html#distributor) | Content cluster | dynamically allocated in the 19100-19899 range | $VESPA\_HOME/sbin/vespa-distributord-bin | Content layer distributor processes |
| [Cluster controller](../../content/content-nodes.html#cluster-controller) | Content cluster | 19050, plus ports dynamically allocated in the 19100-19899 range | java (...) -jar $VESPA\_HOME/lib/jars/container-disc-jar-with-dependencies.jar | Cluster controller processes, manages state for content nodes |
| [proton](../../content/proton.html) | Content cluster | dynamically allocated in the 19100-19899 range | $VESPA\_HOME/sbin/vespa-proton-bin | Searchnode process, receives queries from the container and returns results from the indexes. Also receives feed and indexes documents |
| [container](../../applications/containers.html) | Container cluster | 8080 | java (...) -jar $VESPA\_HOME/lib/jars/container-disc-with-dependencies.jar | Container running servers, handlers and processing components |
##### System limits
The [startup scripts](admin-procedures.html#vespa-start-stop-restart) checks that system limits are set, failing startup if not. Refer to [vespa-configserver.service](https://github.com/vespa-engine/vespa/blob/master/vespabase/src/vespa-configserver.service.in) and [vespa.service](https://github.com/vespa-engine/vespa/blob/master/vespabase/src/vespa.service.in) for minimum values.
##### Core dumps
Example settings:
```
$ mkdir -p /tmp/cores && chmod a+rwx /tmp/cores
$ echo "/tmp/cores/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern
```
This will write files like _/tmp/cores/core.vespa-proton-bi.1721.localhost.1580387387_.
##### Environment variables
Vespa configuration is set in [application packages](../../basics/applications.html). Some configuration is used to bootstrap nodes - this is set in environment variables. Environment variables are only read at startup.
_$VESPA\_HOME/conf/vespa/default-env.txt_ is read in Vespa start scripts - use this to modify variables ([example](multinode-systems.html#aws-ec2)). Each line has the format `action variablename value` where the items are:
| Item | Description |
| --- | --- |
| action | One of `fallback`, `override`, or `unset`. `fallback` sets the variable if it is unset (or empty). `override` set the value regardless. `unset` unsets the variable. |
| variablename | The name of the variable, e.g. `VESPA_CONFIGSERVERS` |
| value | The rest of the line is the variable's value. |
Refer to the [template](https://github.com/vespa-engine/vespa/blob/master/vespabase/conf/default-env.txt.in) for format.
| Environment variable | Description |
| --- | --- |
| VESPA\_CONFIGSERVERS | A comma-separated list of hosts to run configservers, use fully qualified hostnames. Should always be set to the same value on all hosts in a multi-host setup. If not set, `localhost` is assumed. Refer to [configuration server operations](configuration-server.html). |
| VESPA\_HOSTNAME |
Vespa uses `hostname` for node identity. But sometimes this doesn't work properly, either because that name can't be used to find an IP address which works for connecting to services running on the node, or it's just that the name doesn't agree with what the config server thinks the node's host name is. In this case, override by setting the `VESPA_HOSTNAME`, to be used instead of running the `hostname` command.
Note that `VESPA_HOSTNAME` will be used _both_ when a node identifies itself to the config server _and_ when a service on that node registers a network connection point that other services can connect to.
An error message with "hostname detection failed" is emitted if the `VESPA_HOSTNAME` isn't set and the hostname isn't usable. If `VESPA_HOSTNAME` is set to something that cannot work, an error with "hostname validation failed" is emitted instead.
|
| VESPA\_CONFIG\_SOURCES | Used by libraries like the [Document API](../../writing/document-api-guide.html) to set config server endpoints. Refer to [configuration server operations](configuration-server.html#configuration) for example use. |
| VESPA\_WEB\_SERVICE\_PORT | The port number where REST apis will run, default `8080`. This isn't strictly needed, as the port number can be set for each HTTP server in `services.xml`, but with a big application it can be easier to set the default port number just once. Also note that this needs to be set when starting the _configserver_, since the REST api implementation gets its port number from there. |
| VESPA\_TLS\_CONFIG\_FILE | Absolute path to [TLS configuration file](../../security/mtls). |
| VESPA\_CONFIGSERVER\_JVMARGS | JVM arguments for the config server - see [tuning](../../performance/container-tuning.html#config-server-and-config-proxy). |
| VESPA\_CONFIGPROXY\_JVMARGS | JVM arguments for the config proxy - see [tuning](../../performance/container-tuning.html#config-server-and-config-proxy). |
| VESPA\_LOG\_LEVEL | Tuning of log output from tools, see [controlling log levels](../../reference/operations/log-files.html#controlling-log-levels). |
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Files, Processes, Ports, Environment](#page-title)
- [Directories](#directories)
- [Processes and ports](#processes-and-ports)
- [System limits](#vespa-system-limits)
- [Core dumps](#core-dumps)
- [Environment variables](#environment-variables)
---
### Live-upgrading Vespa
[](/en/operations/self-managed/live-upgrade.html.md "View as Markdown")
#### Live-upgrading Vespa
[](/en/operations/self-managed/live-upgrade.html.md "View as Markdown")
This document describes how to live upgrade a Vespa instance. Use this procedure to upgrade without disruption to read or write traffic.
1. **Before upgrading**
- If upgrading to a **new major version**: Upgrade to the latest version on the current major first, then read the release notes for the new major before progressing.
- If upgrading to a new minor version [you should upgrade to a version that is less than 2 months older than the one you are running](../../learn/releases.html#versions) When upgrading a config server this is verified by checking that the minor version number bump is less than 30, otherwise the config server refuses to start. This behavior can be overridden at your own risk by setting environment variable _VESPA\_SKIP\_UPGRADE\_CHECK=true_ on config servers before upgrading.
- Redundancy: For availability, there must be sufficient capacity to take one node per cluster out of service at the time. If the clusters have redundancy=1, or searchable-copies=1, some data will not be available during the upgrade (reduced coverage).
- To reduce node downtime, download the new Vespa version to all hosts in advance.
2. **Detach the application nodes** Not necessary in Vespa 8, for upgrading between Vespa 7 versions see [Vespa 8 release notes](../../reference/release-notes/vespa8.html#upgrade-procedure).
3. **Upgrade config servers**
- Install the new Vespa version on the config servers and [restart](admin-procedures.html#vespa-start-stop-restart) them one by one. Wait until it is up again, look in vespa log for "Changing health status code from 'initializing' to 'up'" or use [health checks](configuration-server.html#troubleshooting).
- Redeploy and activate the application:
```
$[vespa](../../clients/vespa-cli.html#deployment)prepare && vespa activate
```
- The other nodes in the system will not receive config until they are upgraded to the new version (there will be warnings in vespa log containing "Request callback failed: UNKNOWN\_VESPA\_VERSION" until the node is upgraded). This is to make sure that no new, possibly incompatible, config is served.
4. **Upgrade all other nodes one by one** - for each of the other nodes in the system:
- [Stop services](admin-procedures.html#vespa-start-stop-restart) on the node.
- Install the new Vespa version.
- [Start services](admin-procedures.html#vespa-start-stop-restart) on the node.
- Wait until the node is fully up before doing the next node - metrics/interfaces to be used to evaluate if the next node can be stopped:
- Check if a node is up using [/state/v1/health](../../reference/api/state-v1.html#state-v1-health).
- Check the `vds.idealstate.merge_bucket.pending.average` metric on content nodes. When 0, all buckets are in sync - see [example](../metrics.html).
###### Troubleshooting
See [config server troubleshooting](configuration-server.html#troubleshooting).
Copyright © 2026 - [Cookie Preferences](#)
---
### Monitoring
[](/en/operations/self-managed/monitoring.html.md "View as Markdown")
#### Monitoring
[](/en/operations/self-managed/monitoring.html.md "View as Markdown")
Vespa provides metrics integration with CloudWatch, Datadog and Prometheus / Grafana, as well as a JSON HTTP API.
There are two main approaches to transfer metrics to an external system:
- Have the external system _pull_ metrics from Vespa
- Make Vespa _push_ metrics to the external system
Use the example overview of two nodes running Vespa for where the APIs are set up and how they interact:

- [/metrics/v1/values](#metrics-v1-values) is the node metrics api, and aggregates metrics for processes running on the node.
- [/state/v1/metrics](#state-v1-metrics) is the process metrics api, and exposes all metrics from an individual service - here each node runs a container and a content node.
- [/metrics/v2/values](#metrics-v2-values) is an aggregation of [/metrics/v1/values](#metrics-v1-values), for all nodes. Served on the metrics-proxy port.
- [/prometheus/v1/values](/en/reference/api/prometheus-v1.html#prometheus-v1-values) is the same as [/metrics/v1/values](#metrics-v1-values), in prometheus format. Served on the metrics-proxy port.
- [/prometheus/v1/values](/en/reference/api/prometheus-v1.html#prometheus-v1-values) and [/metrics/v2/values](#metrics-v2-values) are also replicated on the container port, default 8080.
**Note:** refer to the [multinode](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode) and [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) sample applications for a practical example of using the APIs. These apps also include examples for how to find ports used by using [vespa-model-inspect](../../reference/operations/self-managed/tools.html#vespa-model-inspect).
See the [metrics guide](../metrics.html) for how to get a metric using `/metrics/v1/values` and `/prometheus/v1/values`. This guide also documents use of custom metrics and histograms.
##### Metrics proxy
Each Vespa node has a _metrics-proxy_ process running for this API, default port 19092. It aggregates metrics from all processes on the node, and across nodes:
The metrics-proxy normally listens on port 19092 - use [vespa-model-inspect](../../reference/operations/self-managed/tools.html#vespa-model-inspect) to validate.
See the [metrics guide](../metrics.html) for the metrics interfaces hosted by the metrics proxy.
Metric-proxies intercommunicate to build a metric cache served on the internal _applicationmetrics/v1/_ API. This is replicated on the container on _/metrics/v2/values_ for easy access to all metrics for an application.
The metrics-proxy is started by the [config-sentinel](config-sentinel.html) and is not configurable. The metrics-proxy process looks like:
```
$ ps ax | grep admin/metrics/vespa-container
703 ? Sl 0:10 /usr/bin/java
-Dconfig.id=admin/metrics/vespa-container
...
-cp /opt/vespa/lib/jars/jdisc_core-jar-with-dependencies.jar
com.yahoo.jdisc.core.StandaloneMain
file:/opt/vespa/lib/jars/container-disc-jar-with-dependencies.jar
```
##### /state/v1/health
_Per-process_ health status is found at _http://host:port/state/v1/health_

`/state/v1/health` is most commonly used for heartbeating, see the [reference](../../reference/api/state-v1.html#state-v1-health) for details. Example:
```json
{
"status": {
"code": "up",
"message": "Everything ok here"
}
}
```
##### /state/v1/metrics
_Per-process_ metrics are found at _http://host:port/state/v1/metrics_
Internally, Vespa aggregates metrics in the APIs above from the _per-process_ metrics and health APIs. While most users would use the aggregated APIs, the per-process metric APIs could be used for specific cases.
Metrics are reported in snapshots, where the snapshot specifies the time window the metrics are gathered from. Typically, the service will aggregate metrics as they are reported, and after each snapshot period, a snapshot is taken of the current values, and they are reset. Using this approach, min and max values are tracked, and enables values like 95% percentile for each complete snapshot period.
Refer to the [reference](../../reference/api/state-v1.html#state-v1-metrics) for details.
Vespa supports [custom metrics](../metrics.html#metrics-from-custom-components).
Example:
```json
{
"status" : {
"code" : "up",
"message" : "Everything ok here"
},
"metrics" : {
"snapshot" : {
"from" : 1334134640.089,
"to" : 1334134700.088,
},
"values" : [
{
"name" : "queries",
"description" : "Number of queries executed during snapshot interval",
"values" : {
"count" : 28,
"rate" : 0.4667
},
"dimensions" : {
"chain" : "vespa"
}
},
{
"name" : "hits_per_query",
"description" : "Number of hits returned for queries during snapshot interval",
"values" : {
"count" : 28,
"rate" : 0.4667,
"average" : 128.3,
"min" : 0,
"max" : 1000,
"sum" : 3584,
"last" : 72,
"95percentile" : 849.1,
"99percentile": 672.0,
},
"dimensions" : {
"chain" : "vespa"
}
}
]
}
}
```
A flat list of metrics is returned. Each metric value reported by a component should be a separate metric. For related metrics, prefix metric names with common parts and dot separate the names - e.g. `memory.free` and `memory.virtual`.
###### /metrics/v1/values
This API can be used for monitoring, using products like [Prometheus](#pulling-into-prometheus) and [DataDog](#pulling-into-datadog). The response contains a selected set of metrics from each service running on the node, see the [reference](../../reference/api/metrics-v1.html) for details. Example:
```
$ curl http://localhost:19092/metrics/v1/values
```
```json
{
"services": [
{
"name": "vespa.container",
"timestamp": 1661945852,
"status": {
"code": "up",
"description": "Data collected successfully"
},
"metrics": [
{
"values": {
"memory_virt": 3693178880,
"memory_rss": 1331331072,
"cpu": 2.3794255627932,
"cpu_util": 0.2379425562793
},
"dimensions": {
"metrictype": "system",
"instance": "container",
"clustername": "default",
"vespaVersion": "8.43.64"
}
}
]
}
]
}
```
###### /metrics/v2/values
```
$ curl http://localhost:19092/metrics/v2/values
```
A container service on the same node as the metrics proxy might forward `/metrics/v2/values` on its own port, normally 8080.
`/metrics/v2/values` exposes a selected set of metrics for every service on all nodes for the application. For example, it can be used to [pull Vespa metrics to Cloudwatch](https://github.com/vespa-engine/metrics-emitter/tree/master/cloudwatch) using an AWS lambda function.
The [metrics API](#metrics-v2-values) exposes a [selected set of metrics](https://github.com/DataDog/integrations-extras/blob/master/vespa/metadata.csv) for the whole application, or for a single node, to allow integration with graphing and alerting services.
The response is a `nodes` list with metrics (see example output below), see the [reference](../../reference/api/metrics-v2.html) for details.
```json
{
"nodes": [
{
"hostname": "vespa-container",
"role": "hosts/vespa-container",
"services": [
{
"name": "vespa.container",
"timestamp": 1634127924,
"status": {
"code": "up",
"description": "Data collected successfully"
},
"metrics": [
{
"values": {
"memory_virt": 3685253120,
"memory_rss": 1441259520,
"cpu": 29.1900152827305
},
"dimensions": {
"serviceId": "container"
}
},
{
"values": {
"jdisc.gc.ms.average": 0
},
"dimensions": {
"gcName": "G1OldGeneration",
"serviceId": "container"
}
},
```
###### /prometheus/v1/values
Vespa provides a _node metrics API_ on each _node_ at _http://host:port/prometheus/v1/values_
Port and content is the same as _/metrics/v1/values_.
The prometheus API on each node exposes metrics in a text based [format](https://prometheus.io/docs/instrumenting/exposition_formats/) that can be scraped by [Prometheus](https://prometheus.io/docs/introduction/overview/). See below for a Prometheus / Grafana example.
##### Pulling metrics from Vespa
All pull-based solutions use Vespa's [metrics API](#metrics-v2-values), which provides metrics in JSON format, either for the full system or for a single node. The polling frequency should be limited to max once every 30 seconds as more frequent polling would not give increased granularity but only lead to unnecessary load on your systems.
| Service | Description |
| --- | --- |
| CloudWatch |
Metrics can be pulled into CloudWatch from both [Vespa Cloud](https://cloud.vespa.ai/) and self-hosted Vespa. The recommended solution is to use an AWS lambda function, as described in [Pulling Vespa metrics to Cloudwatch](https://github.com/vespa-engine/metrics-emitter/tree/master/cloudwatch).
|
| Datadog |
The Vespa team has created a Datadog Agent integration to allow real-time monitoring of Vespa in Datadog. The [Datadog Vespa](https://docs.datadoghq.com/integrations/vespa/) integration is not packaged with the agent, but is included in Datadog's [integrations-extras](https://github.com/DataDog/integrations-extras) repository. Clone it and follow the steps in the [README](https://github.com/DataDog/integrations-extras/blob/master/vespa/README.md).
**Note:** The Datadog Agent integration currently works for self-hosted Vespa only.
|
| Prometheus |
Vespa exposes metrics in a text based [format](https://prometheus.io/docs/instrumenting/exposition_formats/) that can be scraped by [Prometheus](https://prometheus.io/docs/introduction/overview/). For [Vespa Cloud](https://cloud.vespa.ai/), append _/prometheus/v1/values_ to your endpoint URL. For self-hosted Vespa the URL is: _http://\:\/prometheus/v1/values_, where the _port_ is the same as for searching, e.g. 8080. Metrics for each individual host can also be retrieved at `http://host:19092/prometheus/v1/values`.
See the below for a Prometheus / Grafana example.
|
##### Pushing metrics to CloudWatch
**Note:** This method currently works for self-hosted Vespa only.
This is presumably the most convenient way to monitor Vespa in CloudWatch. Steps / requirements:
1. An IAM user or IAM role that only has the _putMetricData_ permission.
2. Store the credentials for the above user or role in a [shared credentials file](https://docs.aws.amazon.com/ses/latest/dg/create-shared-credentials-file.html) on each Vespa node. If a role is used, provide a mechanism to keep the credentials file updated when keys are rotated.
3. Configure Vespa to push metrics to CloudWatch - example configuration for the [admin](/en/reference/applications/services/admin.html) section in _services.xml_:
```xml
```
This configuration sends the default set of Vespa metrics to the CloudWatch namespace `my-vespa-metrics` in the `us-east-1` region. Refer to the [metric list](https://github.com/DataDog/integrations-extras/blob/master/vespa/metadata.csv) for `default` metric set.
##### Monitoring with Grafana
Follow these steps to set up monitoring with Grafana for a Vespa instance. This guide builds on the [quick start](../../basics/deploy-an-application-local.html) by adding three more Docker containers and connecting these in the Docker _monitoring_ network:

1. **Run the Quick Start:**
2. **Create a network and add the _vespa_ container to it:**
3. **Launch Prometheus:**
4. **Launch Grafana:**
5. **Build the Random Data Feeder:**
6. **Run the Random Data Feeder:**
7. **Check the updated Grafana metrics:**
8. **Remove containers and network:**
##### Histograms
Metric histograms is supported for [Gauge](https://javadoc.io/doc/com.yahoo.vespa/container-disc/latest/com/yahoo/metrics/simple/Gauge.html) metrics. Create the metric like in [album-recommendation-java](https://github.com/vespa-engine/sample-apps/tree/master/album-recommendation-java), adding the histogram:
```
public HitCountSearcher(MetricReceiver receiver) {
this.hitCountMetric = receiver.declareGauge(EXAMPLE_METRIC_NAME, Optional.empty(),
new MetricSettings.Builder().histogram(true).build());
}
```
The histograms for the last five minutes of logged data are available as CSV per dimension at [/state/v1/metrics/histograms](../../reference/api/state-v1.html#state-v1-metrics-histograms). Example output:
```
#### start of metric hits_per_query, dimensions: { "chain": "metalchain" }
"Value","Percentile","TotalCount","1/(1-Percentile)"
1.00,0.000000000000,1,1.00
1.00,1.000000000000,1,Infinity
#### end of metric hits_per_query, dimensions: { "chain": "metalchain" }
#### start of metric example_hitcounts, dimensions: { "query_language": "en" }
"Value","Percentile","TotalCount","1/(1-Percentile)"
1.00,0.000000000000,1,1.00
1.00,1.000000000000,1,Infinity
#### end of metric example_hitcounts, dimensions: { "query_language": "en" }
#### start of metric query_latency, dimensions: { "chain": "metalchain" }
"Value","Percentile","TotalCount","1/(1-Percentile)"
5.69,0.000000000000,1,1.00
5.69,1.000000000000,1,Infinity
#### end of metric query_latency, dimensions: { "chain": "metalchain" }
#### start of metric totalhits_per_query, dimensions: { "chain": "metalchain" }
"Value","Percentile","TotalCount","1/(1-Percentile)"
1.00,0.000000000000,1,1.00
1.00,1.000000000000,1,Infinity
#### end of metric totalhits_per_query, dimensions: { "chain": "metalchain" }
```
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Monitoring](#page-title)
- [Metrics proxy](#metrics-proxy)
- [/state/v1/health](#state-v1-health)
- [/state/v1/metrics](#state-v1-metrics)
- [/metrics/v1/values](#metrics-v1-values)
- [/metrics/v2/values](#metrics-v2-values)
- [/prometheus/v1/values](#prometheus-v1-values)
- [Pulling metrics from Vespa](#pulling-metrics-from-vespa)
- [Pushing metrics to CloudWatch](#pushing-metrics-to-cloudwatch)
- [Monitoring with Grafana](#monitoring-with-grafana)
- [Histograms](#histograms)
---
### Multinode systems
[](/en/operations/self-managed/multinode-systems.html.md "View as Markdown")
#### Multinode systems
[](/en/operations/self-managed/multinode-systems.html.md "View as Markdown")
A Vespa _system_ consists of one or more stateless and stateful clusters configured by an application package. A Vespa system is configured and managed through an admin cluster as shown below.

All nodes of a Vespa system have the same software installed. Which processes are started on each node and how they are configured is determined by the admin cluster from the specification given in [services.xml](../../reference/applications/services/services.html) in the application package.
##### Creating a multinode system from a sample application
To create a fully functional production ready multinode system from a single-node sample application, follow these steps (also see [next steps](#next-steps)):
1. Add an [admin cluster](../../reference/applications/services/admin.html) in services.xml:
```xml
```
2. Install the Vespa packages or the _vespaengine/vespa_ Docker image on all the nodes.
3. Run
```
$ echo "override VESPA_CONFIGSERVERS [configserver-hostnames]" >> $VESPA_HOME/conf/vespa/default-env.txt
```
where `[configserver-hostnames]` is replaced by the full hostname of the config server (or a comma-separated list if multiple).
4. Add these nodes to the container and content clusters by adding more `node` tags in _services.xml_.
5. Add the same nodes to _hosts.xml_.
6. Start Vespa on the nodes
See below for AWS examples. Refer to [configuration server operations](configuration-server.html) for troubleshooting.
##### AWS EC2
The following is a procedure to set up a multinode application on _AWS EC2_ instances. Please run the procedure in [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) first, to get familiar with the different Vespa concepts before running the AWS procedure below. This procedure will use the name number of hosts, 10, and set up the same application.
**Important:** Note the use of `sudo`. The Vespa start scripts will modify the environment (directories, system limits), requiring root access - refer to [vespa-start-configserver](../../reference/operations/self-managed/tools.html#vespa-start-configserver) and [vespa-start-services](../../reference/operations/self-managed/tools.html#vespa-start-services). After the environment setup, Vespa is run as the `vespa` user.
**Note:** The procedure below is a bare minimum, for educational purposes. Make sure to use AWS instance types suitable for the application load, and implement security mechanisms of choice.
Can [AWS Auto Scaling](https://aws.amazon.com/autoscaling/) be used? Read the [autoscaling Q/A](#autoscaling).
###### Node setup
- Provision nodes:
- Find AMI at [CentOS AWS AMI Cloud Images](https://centos.org/download/aws-images/) - this procedure is tested with _CentOS Stream 8 us-east-1 x86\_64 ami-0ee70e88eed976a1b_ and vespa-8.30.50.
- Use minimum _t2.medium_ instances.
- Let AWS create a security group for the nodes, or use an existing one.
- Make sure to check for SSH traffic, for host login.
- Launch 10 instances - the 3 first will be Vespa config server nodes, the 7 last Vespa nodes. Write down private / public hostnames. The private names are used in Vespa configuration, the public names for login to check status. To find a hostname, click the instance and copy hostname from _Private IP DNS name (IPv4 only)_ and _Public IPv4 DNS_. Create a table like:
| type | Private IP DNS name (IPv4 only) | Public IPv4 DNS |
| --- | --- | --- |
| configserver | ip-10-0-1-234.ec2.internal | ec2-3-231-33-190.compute-1.amazonaws.com |
| configserver | ip-10-0-1-154.ec2.internal | ec2-3-216-28-201.compute-1.amazonaws.com |
| configserver | ip-10-0-0-88.ec2.internal | ec2-34-230-33-42.compute-1.amazonaws.com |
| services | ip-10-0-1-95.ec2.internal | ec2-44-192-98-165.compute-1.amazonaws.com |
| services | ip-10-0-0-219.ec2.internal | ec2-3-88-143-47.compute-1.amazonaws.com |
| services | ip-10-0-0-28.ec2.internal | ec2-107-23-52-245.compute-1.amazonaws.com |
| services | ip-10-0-0-67.ec2.internal | ec2-54-198-251-100.compute-1.amazonaws.com |
| services | ip-10-0-1-84.ec2.internal | ec2-44-193-84-85.compute-1.amazonaws.com |
| services | ip-10-0-0-167.ec2.internal | ec2-54-224-15-163.compute-1.amazonaws.com |
| services | ip-10-0-1-41.ec2.internal | ec2-44-200-227-127.compute-1.amazonaws.com |
- Security group setup:
- Click the Security Group for the nodes just provisioned (under the security tab), then _Edit inbound rules_. Add _All TCP_ for port range 0-65535, specifying the name of the current Security Group as the Source. This lets the hosts communicate with each other.
- Host login example, without ssh-agent:
```
$ SSH_AUTH_SOCK=/dev/null ssh -i mykeypair.pem centos@ec2-3-231-33-190.compute-1.amazonaws.com
```
- On each of the 10 hosts, install Vespa using the [installation procedure](build-install.html#rpms):
```
$ sudo dnf config-manager \
--add-repo https://raw.githubusercontent.com/vespa-engine/vespa/master/dist/vespa-engine.repo
$ sudo dnf config-manager --enable powertools
$ sudo dnf install -y epel-release
$ sudo dnf install -y vespa
$ export VESPA_HOME=/opt/vespa
```
- On all the 10 hosts, set up the environment using the config server host list:
```
$ echo "override VESPA_CONFIGSERVERS" \
"ip-10-0-1-234.ec2.internal,ip-10-0-1-154.ec2.internal,ip-10-0-0-88.ec2.internal" \
| sudo tee -a $VESPA_HOME/conf/vespa/default-env.txt
```
It is required that all nodes, both config server and Vespa nodes, have the same setting for `VESPA_CONFIGSERVERS`.
###### Config server cluster setup
- Start the 3-node config server cluster:
```
$ sudo systemctl start vespa-configserver
```
- Verify the config cluster is running - on one of the config server nodes:
```
$ for configserver in \
ip-10-0-1-234.ec2.internal \
ip-10-0-1-154.ec2.internal \
ip-10-0-0-88.ec2.internal; \
do curl -s http://$configserver:19071/state/v1/health | head -5; done
{
"time" : 1660034756595,
"status" : {
"code" : "up"
},
{
"time" : 1660034756607,
"status" : {
"code" : "up"
},
{
"time" : 1660034756786,
"status" : {
"code" : "up"
},
```
A successful config server start will log an entry like:
```
$ $VESPA_HOME/bin/vespa-logfmt | grep "Application config generation"
[2022-08-09 08:29:38.684] INFO : configserver
Container.com.yahoo.container.jdisc.ConfiguredApplication
Switching to the latest deployed set of configurations and components.
Application config generation: 0
```
Do not continue setup before the config server cluster is successfully started. See the video: [Troubleshooting startup - multinode](https://www.youtube.com/embed/BG7XZmXpIzo) and read [config server start sequence](configuration-server.html#start-sequence).
- Start Vespa services on the 3 config server nodes - this starts basic Vespa services like log forwarding:
```
$ sudo systemctl start vespa
```
_$VESPA\_HOME/logs/vespa/vespa.log_ will now contain messages for `APPLICATION_NOT_LOADED`, this is normal until an application is deployed (next section).
###### Configure application
- Configure the sample application - on one of the config server nodes:
```
$ sudo dnf install -y git zip
$ git clone https://github.com/vespa-engine/sample-apps.git && \
cd sample-apps/examples/operations/multinode-HA
```
- Edit _hosts.xml_ - replace the _nodeX.vespanet_ names. Let the 3 first hosts be the config server hosts above, the 7 rest the Vespa hosts - example:
```xml
node0
node1
node2
node3
node4
node5
node6
node7
node8
node9
```
- Deploy the application:
```
$ zip -r - . -x "img/*" "scripts/*" "pki/*" "tls/*" README.md .gitignore | \
curl --header Content-Type:application/zip --data-binary @- \
http://localhost:19071/application/v2/tenant/default/prepareandactivate
```
Expected output:
```json
{
"log": [],
"tenant": "default",
"url": "http://localhost:19071/application/v2/tenant/default/application/default/environment/prod/region/default/instance/default",
"message": "Session 2 for tenant 'default' prepared and activated.",
"configChangeActions": {
"restart": [],
"refeed": [],
"reindex": []
}
}
```
###### Vespa nodes setup
- Start Vespa on the 7 hosts:
```
$ sudo systemctl start vespa
```
- Validate the installation. Use the [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) steps to check the health interfaces on all 10 nodes. Note that in this guide, the ports are not mapped through a Docker container, so the native Vespa ports should be used - e.g. for nodes 4 to 7 (see illustration below):
```
$ curl http://localhost:8080/state/v1/health
{
"time" : 1660038306465,
"status" : {
"code" : "up"
},
```
Refer to the sample application ports: 
###### Terminate instances
Remember to terminate the instances in the AWS console after use.
###### AWS EC2 singlenode
This is a variant of the multinode install, using only one host, running both a config server and the other Vespa services on the same node.
- Provision a node, minimum a _t2.large_. Get its hostname for use in `VESPA_CONFIGSERVERS`:
```
$ hostname
```
- Install Vespa:
```
$ sudo dnf config-manager \
--add-repo https://raw.githubusercontent.com/vespa-engine/vespa/master/dist/vespa-engine.repo
$ sudo dnf config-manager --enable powertools
$ sudo dnf install -y epel-release
$ sudo dnf install -y vespa
$ export VESPA_HOME=/opt/vespa
$ echo "override VESPA_CONFIGSERVERS ip-172-31-95-248.ec2.internal" | \
sudo tee -a $VESPA_HOME/conf/vespa/default-env.txt
```
- Get a sample application:
```
$ sudo dnf install -y git zip
$ git clone https://github.com/vespa-engine/sample-apps.git && cd sample-apps/album-recommendation
```
- Start the config server, check health port after a few seconds:
```
$ sudo systemctl start vespa-configserver
$ curl http://localhost:19071/state/v1/health | head -5
```
- Deploy the sample application:
```
$ zip -r - . -x "img/*" "scripts/*" "pki/*" "tls/*" README.md .gitignore | \
curl --header Content-Type:application/zip --data-binary @- \
http://localhost:19071/application/v2/tenant/default/prepareandactivate
```
- Start Vespa, check container node health after some seconds:
```
$ sudo systemctl start vespa
$ curl http://localhost:8080/state/v1/health | head -5
```
- Remember to terminate the instances in the AWS console after use.
##### AWS ECS
The following is a procedure to set up a multinode application on [AWS ECS](https://us-east-1.console.aws.amazon.com/ecs) instances. Please run the procedure in [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) first, to get familiar with the different Vespa concepts before running the AWS procedure below. This procedure will use the name number of host, 10, and set up the same application. Running the [EC2 procedure](#aws-ec2) above can also be helpful, this procedure has a similar structure.
###### Create a 10-node ECS cluster
- Log in to AWS and the EC2 Container Service. Click _Clusters \> Create Cluster \> EC2 Linux + Networking \> Next step_, using the defaults and:
| Cluster name | vespa |
| EC2 instance type | t2.medium |
| Number of instances | 10 |
| Key pair | _Select or create your keypair_ |
| Security group inbound rules - port range | 0 - 65535 |
- Click _Create_, wait for the tasks to succeed, then _View Cluster_ - it should say _Registered container instances: 10_ in ACTIVE state.
###### Configure ECS instances
- Click the _ECS Instances tab_ - this should list 10 container instances.
- Select the 3 first Container Instance checkboxes, then _Actions \> View/Edit attributes_.
- Click _Add attribute_. Set `Name=type` and `Value=configserver`, click the green checkbox on the right, then _Close_.
- Select the next 7 Container instance checkboxes, then _Actions \> View/Edit attributes_.
- Click _Add attribute_. Set `Name=type` and `Value=services`, click the green checkbox on the right, then _Close_.
- Write down private / public hostnames and create a table like in the [EC2 procedure](#node-setup) The private names are used in Vespa configuration, the public names for login to check status. To find a hostname, click _ECS Instance \> Instance ID_ and copy hostname from _Private IP DNS name (IPv4 only)_ and _Public IPv4 DNS_.
###### Start the config server task
- Click _Task Definitions \> Create new Task Definition \> EC2 \> Next step_.
- Click _Configure via JSON_ and replace the content with (note the comma-separated hostnames of the config servers addresses):
```json
{
"networkMode": "host",
"containerDefinitions": [
{
"name": "configserver",
"environment": [
{
"name": "VESPA_CONFIGSERVERS",
"value": "ip-10-0-1-234.ec2.internal,ip-10-0-1-154.ec2.internal,ip-10-0-0-88.ec2.internal"
}
],
"image": "vespaengine/vespa",
"privileged": true,
"memoryReservation": 1024
}
],
"placementConstraints": [
{
"expression": "attribute:type == configserver",
"type": "memberOf"
}
],
"family": "configserver"
}
```
- Click _Save \> Create_.
- Choose _Actions -\> Run task_ and configure:
| Launch type | EC2 |
| Cluster | vespa |
| Number of tasks | 3 |
| Placement templates | One Task Per Host |
- Click _Run Task_.
- Validate that the config servers started successfully - use the same procedure as for [EC2 instances](#config-server-cluster-setup), checking _/state/v1/health_. Do not continue before successfully validating this:
```
$ ssh -i mykeypair.pem ec2-user@ec2-3-231-33-190.compute-1.amazonaws.com \
curl -s http://localhost:19071/state/v1/health | head -5
{
"time" : 1660635645783,
"status" : {
"code" : "up"
},
```
###### Configure application - ECS
- Log into a config server:
```
$ ssh -i mykeypair.pem ec2-user@ec2-3-231-33-190.compute-1.amazonaws.com
```
- Download the multinode-HA sample application:
```
$ sudo yum -y install git zip
$ git clone https://github.com/vespa-engine/sample-apps.git && \
cd sample-apps/examples/operations/multinode-HA
```
- Modify _hosts.xml_ using the internal DNS hostnames - this step is the same as for [EC2 instances](#configure-application)
- Deploy the application:
```
$ zip -r - . -x "img/*" "scripts/*" "pki/*" "tls/*" README.md .gitignore | \
curl --header Content-Type:application/zip --data-binary @- \
http://localhost:19071/application/v2/tenant/default/prepareandactivate
```
###### Start the services tasks
- Click _Task Definitions \> Create new Task Definition \> EC2 \> Next step_.
- Click _Configure via JSON_ and replace the content with (using the same 3 config server internal DNS names):
```json
{
"networkMode": "host",
"containerDefinitions": [
{
"name": "services",
"environment": [
{
"name": "VESPA_CONFIGSERVERS",
"value": "ip-10-0-1-234.ec2.internal,ip-10-0-1-154.ec2.internal,ip-10-0-0-88.ec2.internal"
}
],
"image": "vespaengine/vespa",
"command": [
"services"
],
"privileged": true,
"memoryReservation": 1024
}
],
"placementConstraints": [
{
"expression": "attribute:type == services",
"type": "memberOf"
}
],
"family": "services"
}
```
- Click _Save \> Create_. Note the `"command": ["services"]`. See [controlling which services to start](docker-containers.html#controlling-which-services-to-start) for details, this starts _services_ only - the start script starts both the _configserver_ and _services_ if given no arguments - this is used for the config server above. For these 7 nodes, `services` is given as an argument to the start script to only start Vespa services.
- Choose _Actions \> Run task_ and configure:
| Launch type | EC2 |
| Cluster | vespa |
| Number of tasks | 7 |
| Placement templates | One Task Per Host |
- Click _Run Task_.
- Validate startup. This step is the same as for [EC2 instances](#vespa-nodes-setup), e.g. for nodes running a Vespa container the port is 8080:
```
$ ssh -i mykeypair.pem ec2-user@ec2-3-88-143-47.compute-1.amazonaws.com \
curl -s http://localhost:8080/state/v1/health | head -5
{
"time" : 1660652648442,
"status" : {
"code" : "up"
},
```
###### Terminate cluster
- Remember to delete the cluster in the AWS console after use.
##### Log collection
Logs are automatically collected from all nodes in real time to the admin node listed as `adminserver`. To view log messages from the system, run [vespa-logfmt](../../reference/operations/self-managed/tools.html#vespa-logfmt) on this node.
##### Making changes to live systems
To change the system, deploy the changed application to the admin cluster. The admin cluster will automatically change the participating nodes as necessary. It is safe to do this while serving live query and write traffic. In some cases the admin cluster will report that some processes must be restarted to make the change effective. To avoid query or write traffic disruption, such restarts must be done on one node at the time, waiting until the node is fully up before restarting the next one.
##### Multiple proton processes
A content cluster can have multiple schemas. There is another way to distribute load over hosts, by mapping multiple content clusters to the same hosts:
```xml
```
Observe that both clusters use `node1`. This is a non-recommended configuration, as it runs multiple [proton](../../content/proton.html) processes per node. To reduce interference between the processes in this case, virtualize the host into more nodes. One can use [containers or VMs](docker-containers.html) to do this:

**Important:** Vespa's features for overload handling, like [feed-block](../../writing/feed-block.html), requires that only one proton process is running on the node.
##### Autoscaling
A common question is, _"Can [AWS Auto Scaling](https://aws.amazon.com/autoscaling/) be used?"_ That is a difficult question to answer, here is a transcript from the [Vespa Slack](https://slack.vespa.ai):
> I have a question about deployment. I set up cluster on two AWS auto-scaling groups (config & services) based on [multinode-systems.html#aws-ec2](#aws-ec2). But if one of instances was replaced by auto-scaling group, I need manually update hosts.xml file, zip it and deploy new version of the app. I'm thinking about automation of this process by Cloudwatch & Lambda... I wonder if there is some node-discovery mechanism which can e.g. check instances tags and update hosts config based on it?
First, you see in [aws-ec2](#aws-ec2) that there are two types of hosts, `configserver` and `services`. configserver setup / operations is documented at [configuration server operations](configuration-server.html). This must be set up first. This is backed by an [Apache ZooKeeper](https://zookeeper.apache.org/) cluster, so should be 1 or 3 nodes large. In our own clusters in Yahoo, we do not autoscale configserver clusters, there is no need - we use 3. If that is too many, use 1. So this question is easy - do not autoscale configservers.
For the services nodes, observe that there are two kinds of nodes - stateless containers and stateful content nodes - see the [overview](../../learn/overview.html). In any way, you will want to manage these differently - the stateless nodes are more easily replaced / increased / shrunk, by changing _services.xml_ and _hosts.xml_. It is doable to build an autoscaling service for the stateless nodes, but you need to make sure to use the right metrics for your autoscaling code, and integrate the deploy-automation with the other deployments (say schema modifications).
A much harder problem is autoscaling the stateful nodes - these are the nodes with the indexes and data. See [elasticity](../../content/elasticity.html) - adding a node + data redistribution can take hours, and the node's load will increase during redistribution. Building autoscaling here is very difficult to do safely and efficient.
Nothing of this is impossible, and it is actually implemented at [cloud.vespa.ai/autoscaling](https://cloud.vespa.ai/#autoscaling) - but it is a difficult feature to get right.
So, my recommendation is starting with a static set of hosts, like in [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) - and in parallel try out [cloud.vespa.ai/en/free-trial](https://cloud.vespa.ai/en/free-trial) with autoscaling experiments using your data and use cases.
Autoscaling can save money, but before going there, it is wise to read about [performance](/en/performance/) and optimize resources using a static node set (or use the sizing suggestions from the Vespa Cloud Console). I.e., get the node resources right first, then consider if autoscaling node count for your load patterns makes sense.
##### Next steps
- [Multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) is a high-availability multi-node template - use this as a basis for the final configuration.
- The [multinode](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode) sample application is a useful for experimenting with node state transitions.
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Multinode systems](#page-title)
- [Creating a multinode system from a sample application](#creating-a-multinode-system-from-a-sample-application)
- [AWS EC2](#aws-ec2)
- [Node setup](#node-setup)
- [Config server cluster setup](#config-server-cluster-setup)
- [Configure application](#configure-application)
- [Vespa nodes setup](#vespa-nodes-setup)
- [Terminate instances](#terminate-instances)
- [AWS EC2 singlenode](#aws-ec2-singlenode)
- [AWS ECS](#aws-ecs)
- [Create a 10-node ECS cluster](#create-a-10-node-ecs-cluster)
- [Configure ECS instances](#configure-ecs-instances)
- [Start the config server task](#start-the-config-server-task)
- [Configure application - ECS](#configure-application-ecs)
- [Start the services tasks](#start-the-services-tasks)
- [Terminate cluster](#terminate-cluster)
- [Log collection](#log-collection)
- [Making changes to live systems](#making-changes-to-live-systems)
- [Multiple proton processes](#multiple-proton-processes)
- [Autoscaling](#autoscaling)
- [Next steps](#next-steps)
---
### Node and network setup
[](/en/operations/self-managed/node-setup.html.md "View as Markdown")
#### Node and network setup
[](/en/operations/self-managed/node-setup.html.md "View as Markdown")
Vespa is composed of services that communicate and interact with each other. These services can be partitioned onto any amount of actual hardware for scaling, or they can all coexist on a single environment for development. To achieve this flexibility, some requirements must be met for the environment where the services will run.
##### Node
A _node_ in this context is the environment where some Vespa services are running. This can be an actual machine like a server in a datacenter, or a laptop for development and testing of Vespa configuration. It can also be a Virtual Machine or a Docker container, so one can run multiple nodes on a single piece of hardware.
The different Vespa services that run on nodes will mostly communicate with each other via the network. This means that all nodes must have an IP address and have network connectivity to all other nodes. Both IPv4 and IPv6 protocols are supported. Note that the same framework is used even when running the entire Vespa stack on a single node.
##### Memory settings
In the [getting started guides](../../basics/deploy-an-application-local.html) and [sample applications](https://github.com/vespa-engine/sample-apps), memory settings are always the minimum to run the guides. This to make it easy to set up and explore Vespa features. The [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) application has examples for even tighter settings, just to be able to test a larger application on a small host.
It is important to note that these are not the recommended Vespa memory settings. Finding the optimal node configuration is covered in the [performance guides](../../performance/index.html). There are many ways to configure, feed and use Vespa, it is not possible to have a general, recommended setting.
**Important:** Vespa is a multiprocess application, and can be configured to run multiple services per node - see [config sentinel](config-sentinel.html). Out-of-memory can cause a range of problems hard to diagnose - the Vespa team recommends testing with larger nodes in these cases.
As a rule of thumb, start with an 8 GB node, just to make sure the application is functionally correct - then optimize later.
Review system defaults. A common issue is too low default for `vm.max_map_count` which makes apps run into memory mapping assertions, especially if the [paged](../../content/attributes.html#paged-attributes) option has been enabled for a lot of attribute fields - symptoms of this can be like:
```
vespalib::alloc::MemoryAllocator::PtrAndSize vespalib::alloc::MmapFileAllocator::alloc(size_t) const: Assertion buf!=MAP_FAILED
```
```
'terminate called after throwing an instance of 'std::runtime_error' 'what(): mmap of file '/opt/vespa/var/db/vespa/search/cluster.abc/n12/documents/xyz/0.ready/index/index.fusion.43/field123/boolocc.bdat' with flags '1' failed with error: 'Cannot allocate memory'
```
##### Hostname
When Vespa services are started on a node, the node must identify itself to the configuration system to get configuration (including which services to run). This requires a unique identifier for the node in the config server. Since it is already a requirement that the node has a _hostname_ that the config server knows, Vespa uses the same _hostname_ when a node identifies itself to get its configuration. See [config sentinel](config-sentinel.html) for details.
In order to find the IP address of a node and connect to it, the node must have a _hostname_ that identifies it and which maps to its IP address. Actual machines on a network will usually have a _Fully Qualified Domain Name_ (FQDN) in DNS, which should be used as the host name for this purpose.
Note that it is a _requirement_ that the host name, configured in [hosts.xml](../../reference/applications/hosts.html), can be used to look up the IP address of the node (see workaround using `VESPA_HOSTNAME` below). The configuration server use this host name to create URLs to be used to open network connections to Vespa services running on that node. If the nodes use IP addresses which don't have DNS names, one _must_ have _all_ those IP addresses with corresponding host names in `/etc/hosts` on _all_ nodes in the Vespa installation. We recommend using names that can be used as FQDNs also in this case, in case of moving to using a DNS server instead of publishing `/etc/hosts`.
This means that the node _must_ know its own hostname (FQDN), and be in agreement with the config server about what exactly the host name is. Usually this is achieved by just running the `hostname` command. If `hostname` is set to the FQDN of the node, then everything should Just Work.
As an alternative to modifying `/etc/hosts`, set [VESPA\_HOSTNAME](files-processes-and-ports.html#environment-variables) on the hosts. [vespa-ip-vs-hostname](https://www.jocas.lt/journal/articles/vespa-ip-vs-hostname/) is a great post on how to do this.
##### Simple single-node development environment
When testing a Vespa configuration on a single-node setup, one can usually avoid the setup hassle by overriding the hostname with the value "localhost". Try this command for that purpose:
```
$ echo "override VESPA_HOSTNAME localhost" >> $VESPA_HOME/conf/vespa/default-env.txt
```
Running Java unit tests won't pick up settings in `default-env.txt` and will default to "localhost" if `VESPA_HOSTNAME` isn't set in the environment.
Copyright © 2026 - [Cookie Preferences](#)
---
### Procedure: Change from attribute to index
[](/en/operations/self-managed/procedure-change-attribute-index.html.md "View as Markdown")
#### Procedure: Change from attribute to index
[](/en/operations/self-managed/procedure-change-attribute-index.html.md "View as Markdown")
Changing between `index` and `attribute` is a common field change operation to optimize performance. Use the [reindexing](../reindexing.html) feature to safely migrate data to/from index structures.
Changing from attribute to index can be seen as "drop attribute" and "add index". When the attribute aspect of a field is removed, the field's data is not queryable after deployment. The reindexing process will populate the field's index structure, but this takes time, depending on corpus size.
Another approach is to run with both attribute and index in the transition, keeping data available for queries. The gist of this procedure is to add `index`, run a reindex - then remove `attribute` aspect:
```vespa-schema-language
#### field configuration at start
field artist type string {
indexing: summary | attribute
}
->
#### intermediate step to populate index structure, keeping the data in the attribute
field artist type string {
indexing: summary | attribute | index
match: word
stemming: none
}
->
#### final configuration, migrated to index
field artist type string {
indexing: summary | index
match: word
stemming: none
}
```
**Note:** If the field is used as a filter only (i.e. no ranking), consider adding `rank: filter`, see example in [feature-tuning](../../performance/feature-tuning.html).
##### Procedure
1. Test this using the [quick-start](../../basics/deploy-an-application-local.html), changing the `artist` field to an attribute before running. Also add a [validation override](../../reference/applications/validation-overrides.html) in `src/main/application/validation-overrides.xml`:
2. Run the quick start, stop after feeding documents. Run a query to validate data can be queried:
3. Add index aspect and match/stemming settings to the field, deploy and observe output
4. Wait for the new configuration generation to be activated on the config server(s) - this is normally quite immediate. After that, allow up to 3 minutes for the config servers to set reindexing ready, track this using the `reindexing` endpoint:
5. When ready, deploy again to start reindexing, wait for it to complete (use the loop in previous step):
6. Dumping the index structures now shows artist both in index and attribute, and there is an entry in vespa.log. Verify the query still works:
7. As data is now reindexed into the index data structures, deploy without attribute. (Observe changes to index files, "artist" is now in index only). Test query after restart:
8. Optional: restart Vespa - a restart will reclaim memory from the attribute:
Notes:
- The match/stemming settings above are set to the same at default attribute settings
##### Appendix
To inspect attribute and index data (can be useful when troubleshooting), use [vespa-proton-cmd](../../reference/operations/self-managed/tools.html#vespa-proton-cmd), then list files:
```
$ docker exec vespa vespa-proton-cmd --local triggerFlush
$ docker exec vespa find /opt/vespa/var/db/vespa/search/cluster.music/n0/documents/music/0.ready
```
Copyright © 2026 - [Cookie Preferences](#)
---
### Vespa Scaling Configuration Examples
[](/en/operations/self-managed/sizing-examples.html.md "View as Markdown")
#### Vespa Scaling Configuration Examples
[](/en/operations/self-managed/sizing-examples.html.md "View as Markdown")
This guide provides some example [services.xml](../../reference/applications/services/content.html) files for content clusters using flat or grouped data distribution in self-managed clusters.
Refer to the [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) sample application for how to get started from a deployable multinode starting point. The examples below are trimmed down for readability in the `admin` and `container` sections. See the [appendix](#appendix-hosts-xml) for _hosts.xml_ to use when testing deployments.
##### Flat Distribution
Flat (single group) distribution with [min-redundancy](../../reference/applications/services/content.html#min-redundancy)=3. Data is distributed and partitioned over 9 nodes and there are 3 replicas of each document, stored on 3 different nodes. Queries are dispatched in parallel to all nodes. In case of a node failure, the remaining nodes will index (make ready) and activate the _not ready_ (stored) copies to restore full search coverage.
```xml
3
```
##### Grouped Distribution
See [sizing search](../../performance/sizing-search.html) more on when to use grouped distribution. When using grouped distribution in an indexed content cluster, the following restrictions apply:
- There can only be a single level of leaf groups under the top group
- The number of leaf groups must be a factor of the _redundancy_
- The [distribution partitions](../../reference/applications/services/content.html#distribution) must be specified such that the redundancy per group is equal
With a low number of nodes per group, it's important to remember that a node failure will cause the data to be re-distributed to the remaining nodes and their memory footprint and disk usage will grow when those nodes start activating the documents originally activated on the failed node. E.g. with 2 nodes per group, the remaining healthy node will start activating all the content, which will cause a 2x memory and disk footprint compared with the ideal state.
The [min-node-ratio-per-group](../../reference/applications/services/content.html#min-node-ratio-per-group) controls the data distribution behavior inside a group in cases of node failures. This sets a lower bound on the ratio of nodes within groups that must be online and accepting feed and query traffic, before the entire group is automatically taken out of service from both feed and search/serving. Once number of nodes in the group have been restored, and ideal state has been achieved, the group will be automatically set in service.
##### 9 nodes, 3 groups with 3 nodes per group
This example has 3 groups and each group index all the documents over the 3 nodes in the group. With 3 groups there are 3 replicas in total of each document, and each replica is indexed and active. Losing a node does not reduce search coverage.
```xml
3
```
##### 9 nodes, 9 groups with 1 node per group
This example has 9 groups and each group index all the documents on a single node. With 9 groups there are 9 replicas in total of each document, and each replica is indexed and active. Losing a node does not reduce search coverage. With a single node, indexing throughput is limited by the single node performance, as all data needs to go all nodes.
```xml
9
```
##### Serving Availability Tuning
When using flat distribution, _soft failing nodes_ is a challenge for serving with high availability and low latency. Soft failing nodes are nodes which answers health checks from [cluster controllers](../../content/content-nodes.html) and search container dispatch health checks, but still experiences issues which impacts serving latency (e.g. cpu frequency throttling due to thermal heating, memory corruptions and so forth). In a cluster with a flat distribution, the slowest node determines the latency, as the query request is dispatched to all content nodes in parallel. The probability of a soft failing node increases with the number of nodes used to distribute the data over.
Use [adaptive coverage timeout](../../reference/applications/services/content.html#coverage) to prevent slow soft failing nodes to impact availability. This allows the dispatcher to stop waiting for the slowest node(s). See also [graceful search degradation](../../performance/graceful-degradation.html).
Grouped distributions are less impacted by soft failing nodes in general, as queries are dispatched to one group at a time using a [dispatch policy](../../reference/applications/services/content.html#dispatch-policy). The _adaptive_ policy takes group latency into account when deciding which group the query request should be routed to.
##### Changing Group Configuration
It is easy to change the group topology without service disruption, with a few caveats - read more in [elasticity](../../content/elasticity.html#changing-topology).
##### Appendix: hosts.xml
```xml
node0
node1
node2
node3
node4
node5
node6
node7
node8
```
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Vespa Scaling Configuration Examples](#page-title)
- [Flat Distribution](#flat-distribution)
- [Grouped Distribution](#grouped-distribution)
- [9 nodes, 3 groups with 3 nodes per group](#9-nodes-3-groups-with-3-nodes-per-group)
- [9 nodes, 9 groups with 1 node per group](#9-nodes-3-groups-with-1-node-per-group)
- [Serving Availability Tuning](#serving-availability-tuning)
- [Changing Group Configuration](#changing-group-configuration)
- [Appendix: hosts.xml](#appendix-hosts-xml)
---
### Service location broker - slobrok
[](/en/operations/self-managed/slobrok.html.md "View as Markdown")
#### Service location broker - slobrok
[](/en/operations/self-managed/slobrok.html.md "View as Markdown")
Slobrok is an acronym for _Service Location Broker_, and it is a name service used in Vespa. The service listens on a specific port - use [vespa-model-inspect](../../reference/operations/self-managed/tools.html#vespa-model-inspect) to find the `Slobrok` service's port number.
Slobrok is running by default on the administration node as well as one or two other random nodes for redundancy. Best practise for a multi-node, high-availability application is found in the [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA) sample application. In this application, slobrok instances are hosted on nodes running [config servers](configuration-server.html). The motivation is, like the config servers, Vespa requires slobrok to be up for services to function. Operating slobrok is the same as config servers, too - three is enough for most applications. As slobrok requires minimal system resources, it does not impact other services running on the same node -\> using config server nodes is ideal.
Clients, like the [Document API](/en/reference/api/api.html), will do lookups on any of the service location broker nodes. Slobrok is not used in the query pipeline. The [cluster-controller](../../content/content-nodes.html#cluster-controller) uses slobrok to evaluate service availability.
The Slobrok process looks like:
```sh
$ ps ax | grep vespa-slobrok
93906 ?? SJ 2:31.52 $VESPA_HOME/sbin/vespa-slobrok -p 19100 -c slobrok.0
```
Copyright © 2026 - [Cookie Preferences](#)
---
### Container GPU setup
[](/en/operations/self-managed/vespa-gpu-container.html.md "View as Markdown")
#### Container GPU setup
[](/en/operations/self-managed/vespa-gpu-container.html.md "View as Markdown")
Vespa supports using GPUs to evaluate ONNX models, as part of its [stateless model evaluation feature](../../ranking/stateless-model-evaluation.html). When running Vespa inside a container engine such as Docker or Podman, special configuration is required to make GPU(s) available inside the container.
The following guide explains how to do this for Nvidia GPUs, using Podman on RHEL8. This should also work on plain Rocky Linux 8.8 and AlmaLinux 8.8 on x86\_64. For other platforms and container engines, see the [Nvidia container toolkit installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). Commands below need to run as root (use `sudo bash` first).
##### Run a script
Fetch and run our script for RHEL8 / x86\_64 and run it as follows:
```
sudo dnf -y install wget
wget https://raw.githubusercontent.com/vespa-engine/docker-image/master/experimental/gpu-setup-rhel8-x86.sh
sh gpu-setup-rhel8-x86.sh
```
This will follow the steps below and check if a sample application is able to utilise the GPU. For more details see the steps below.
##### Configuration steps
1. Check that SELinux is disabled with `getenforce`; edit `/etc/selinux/config` and reboot if necessary. To temporarily avoid SELinux interfering, it's possible to run `setenforce Permissive` instead.
2. Ensure that Nvidia drivers are installed on your **host** where you want to run the `vespaengine/vespa` container image. On RHEL 8 this can be done as follows:
3. Install `nvidia-container-toolkit`. This grants the container engine access to your GPU device(s). On RHEL 8 this can be done as follows:
4. Generate a "Container Device Interface" config:
5. Verify that the GPU device is exposed to the container:
6. Start the Vespa container with the `--device` option:
7. The `vespaengine/vespa` image does not currently include the necessary CUDA libraries by default, due to their large size. These libraries must be installed inside the container manually:
```
podman exec -u 0 -it vespa /bin/bash
dnf -y install dnf-plugins-core
dnf config-manager \
--add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
dnf -y install vespa-onnxruntime-cuda
```
Instead of the above installation of `vespa-onnxruntime-cuda` inside the running container, you might want to build your own container image using the following `Dockerfile` as it avoids having to run the container image with install privileges.
```
FROM vespaengine/vespa
USER root
RUN dnf -y install 'dnf-command(config-manager)'
RUN dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
RUN dnf -y install $(rpm -q --queryformat '%{NAME}-cuda-%{VERSION}' vespa-onnxruntime)
USER vespa
```
Then instead run with your container image name:
```
podman run --detach --name vespa --hostname vespa-container \
--publish 8080:8080 --publish 19071:19071 \
--device nvidia.com/gpu=all \
your-container-image-name
```
8. All Nvidia GPUs on the host should now be available inside the container, with devices exposed at `/dev/nvidiaN`. See [stateless model evaluation](../../ranking/stateless-model-evaluation.html#onnx-inference-options) for how to configure the ONNX runtime to use a GPU for computation. Similar for embedding models using GPU, see [embedder onnx reference](../../reference/rag/embedding.html#embedder-onnx-reference-config).
Copyright © 2026 - [Cookie Preferences](#)
---
### Vespa Support CLI
[](/en/operations/self-managed/vespa-support.html.md "View as Markdown")
#### Vespa Support CLI
[](/en/operations/self-managed/vespa-support.html.md "View as Markdown")
The Vespa Enterprise image includes a built-in command-line tool called `vespa-support`. It is designed to simplify and expedite the process of collecting diagnostic information from your system, making it easier to request and receive support from the Vespa team.
You should run this tool whenever you are:
- Experiencing unexpected behaviour or degraded performance in your Vespa deployment
- Preparing to open or respond to a Vespa support ticket
- Asked by Vespa support to provide diagnostic information
##### Accessing the Tool
`vespa-support` is available in the shell of any running Vespa Enterprise container. To access it, open a shell session into the container using your platform's standard method, then verify the tool is available:
```
$ vespa support --help
```
To confirm the tool version and the container version:
```
$ vespa support version
```
##### Commands
###### vespa support version
Prints the version of the `vespa-support` tool and the Vespa Enterprise container version.
###### vespa support diagnostics sysinfo
Collects host-level system information from the node where the command is run. `--dest-dir` is required. The command will fail if it is not provided.
```
$ vespa support diagnostics sysinfo --dest-dir
```
###### vespa support diagnostics application
Collects the currently deployed Vespa application state. `--dest-dir` is required. The command will fail if it is not provided.
```
$ vespa support diagnostics application --dest-dir
```
##### Options
The following options apply to all `diagnostics` subcommands.
| Option | Required | Default | Description |
| --- | --- | --- | --- |
| `--dest-dir` | Yes | — | Directory where the diagnostic output files will be written. |
| `--config-server-host` | No | Auto-detected | Host address of the Vespa config server. If not set, the tool resolves it automatically from the environment. |
| `--config-server-port` | No | 19071 | Port of the Vespa config server. |
| `--timeout-secs` | No | 60 | Timeout in seconds for operations that contact the config server. |
##### What Is Collected
###### sysinfo
The output is written as a ZIP file named `diag-sysinfo.zip` in the specified destination directory. It includes the following information about the host:
- Vespa version
- Hostname, operating system, kernel version, and CPU architecture
- CPU model and core count
- Memory usage (total, free, and available)
- Disk usage per mounted filesystem (size, free space, usage percentage)
- Network interfaces, routing table, DNS servers, and firewall rules as seen from within the container
- Host level diagnostics metrics
###### application
The output is written as a ZIP file named `diag-application.zip` in the specified destination directory. It includes the following information about the application:
- The full running Vespa application state
- Application level diagnostics metrics
##### Creating a Support Bundle
The following steps walk you through collecting a complete set of diagnostic files to share with the Vespa support team.
**1. Open a shell session into the Vespa Enterprise container.**
**2. Create a directory to store the output:**
```
$ mkdir /tmp/vespa-diagnostics
```
**3. Collect system information:**
```
$ vespa support diagnostics sysinfo --dest-dir /tmp/vespa-diagnostics
```
**4. Collect application diagnostics:**
```
$ vespa support diagnostics application --dest-dir /tmp/vespa-diagnostics
```
**5. Verify the output:**
```
$ ls /tmp/vespa-diagnostics
diag-sysinfo.zip
diag-application.zip
```
Once complete, copy the ZIP files from the container and attach them to your support ticket.
##### Sharing with Vespa Support
Attach the ZIP files generated in the previous step to your Vespa support ticket. If you have not yet opened a ticket, include the files when submitting your request.
If the files are too large to attach directly, the Vespa support team will provide an alternative upload method.
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Vespa Support CLI](#page-title)
- [Accessing the Tool](#accessing-the-tool)
- [Commands](#commands)
- [vespa support version](#version)
- [vespa support diagnostics sysinfo](#diagnostics-sysinfo)
- [vespa support diagnostics application](#diagnostics-application)
- [Options](#options)
- [What Is Collected](#what-is-collected)
- [sysinfo](#sysinfo-output)
- [application](#application-output)
- [Creating a Support Bundle](#creating-a-support-bundle)
- [Sharing with Vespa Support](#sharing-with-vespa-support)
---
## Applications
### services.xml - 'admin'
[](/en/reference/applications/services/admin.html.md "View as Markdown")
#### services.xml - 'admin'
[](/en/reference/applications/services/admin.html.md "View as Markdown")
Reference documentation for `` in [services.xml](services.html). Find a working example of this configuration in the sample application _multinode-HA_[services.xml](https://github.com/vespa-engine/sample-apps/blob/master/examples/operations/multinode-HA/services.xml).
```
admin [version][adminserver [hostalias]](#adminserver)[cluster-controllers](#cluster-controllers)[cluster-controller [hostalias, baseport, jvm-options, jvm-gc-options]](#cluster-controller)[configservers](#configservers)[configserver [hostalias, baseport]](#configserver)[logserver [jvm-options, jvm-gc-options]](#logserver)[slobroks](#slobroks)[slobrok [hostalias, baseport]](#slobrok)[monitoring [systemname]](#monitoring)[metrics](#metrics)[consumer [id]](#consumer)[metric-set [id]](#metric-set)[metric [id]](#metric)[cloudwatch [region, namespace]](#cloudwatch)[shared-credentials [file, profile]](#shared-credentials)[logging](#logging)
```
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| version | required | number | |
2.0
|
##### adminserver
The configured node will be the default administration node in your Vespa system, which means that unless configured otherwise all administrative services - i.e. the log server, the configuration server, the slobrok, and so on - will run on this node. Use [configservers](#configservers), [logserver](#logserver), [slobroks](#slobroks) elements if you need to specify baseport or jvm options for any of these services.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| hostalias | required | string | |
|
| baseport | optional | number | |
|
##### cluster-controllers
Container for one or more [cluster-controller](#cluster-controller) elements. When having one or more [content](content.html) clusters, configuring at least one cluster controller is required.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| standalone-zookeeper | optional | true/false | false |
Will by default share the ZooKeeper instance with configserver. If configured to true a separate ZooKeeper instance will be configured and started on the set of nodes where you run cluster controller on. The set of cluster controllers nodes cannot overlap with the set of nodes where config server is running. If this setting is changed from false to true in a running system, all previous cluster state information will be lost as the underlying ZooKeeper changes. Cluster controllers will re-discover the state, but nodes that have been manually set as down will again be considered to be up.
|
##### cluster-controller
Specifies a host on which to run the [Cluster Controller](../../../content/content-nodes.html#cluster-controller) service. The Cluster Controller manages the state of the cluster in order to provide elasticity and failure detection.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| hostalias | required | string | |
|
| baseport | optional | number | |
|
| jvm-options | optional | string | |
|
##### configservers
Container for one or more `configserver` elements.
##### configserver
Specifies a host on which to run the [Configuration Server](/en/operations/self-managed/configuration-server.html) service. If contained directly below `` you may only have one, so if you need to configure multiple instances of this service, contain them within the [``](#configservers) element.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| hostalias | required | string | |
|
| baseport | optional | number | |
|
##### logserver
Specifies a host on which to run the [Vespa Log Server](../../operations/log-files.html#log-server) service. If not specified, the logserver is placed on the [adminserver](#adminserver), like in the [example](https://github.com/vespa-engine/sample-apps/blob/master/examples/operations/multinode-HA/services.xml).
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| hostalias | required | string | |
|
| baseport | optional | number | |
|
| jvm-options | optional | string | |
|
| jvm-gc-options | optional | string | |
|
Example:
```xml
```
##### slobroks
This is a container for one or more `slobrok` elements.
##### slobrok
Specifies a host on which to run the [Service Location Broker (slobrok)](/en/operations/self-managed/slobrok.html) service.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| hostalias | required | string | |
|
| baseport | optional | number | |
|
##### monitoring
Settings for how to pass metrics to a monitoring service - see [monitoring](/en/operations/self-managed/monitoring.html).
```xml
```
| systemname | The name of the application in question in the monitoring system, default is "vespa" |
##### logging
Used for tuning log levels of Java plug-ins. If you (temporarily) need to enable debug logging from some class or package, or if some third-party component is spamming your log with unnecessary INFO level messages, you can turn levels on or off. Example:
```xml
```
Note that tuning also affects sub-packages, so the above would also affect all packages with `org.anotherorg.` as prefix. And if there is a `org.myorg.tricky.package.foo.InternalClass` you will get even "spam" level logging from it!
The default for `levels` is `"all -debug -spam"` and as seen above you can add and remove specific levels.
##### metrics
Used for configuring the forwarding of metrics to graphing applications - add `consumer` child elements. Also see [monitoring](/en/operations/self-managed/monitoring.html). Example:
```xml
```
##### consumer
Configure a metrics consumer. The metrics contained in this element will be exported to the consumer with the given id. `consumer` is a request parameter in [/metrics/v1/values](../../api/metrics-v1.html), [/metrics/v2/values](../../api/metrics-v2.html) and [/prometheus/v1/values](../../api/prometheus-v1.html).
Add `metric` and/or `metric-set` children.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The name of the consumer to export metrics to.
|
##### metric-set
Include a pre-defined set of metrics to the consumer.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The id of the metric set to include. Built-in metric sets are:
- `default`
- `Vespa`
|
##### metric
Configure a metric.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The name of the metric as defined in custom code or in [process metrics api](../../api/state-v1.html#state-v1-metrics)
|
Note that metric id needs to include the metric specific suffix, e.g. _.average_.
In this example, there is one metric added to a custom consumer in addition to the default metric set. Use _&consumer=my-custom-consumer_ parameter for the prometheus endpoint. Also notice the .count suffix, see [process metrics api](../../api/state-v1.html#state-v1-metrics).
The per process metrics api endpoint _/state/v1/metrics_ also includes a description of each emitted metric. The _/state/v1/metrics_ endpoint also includes the metric aggregates (.count, .average, .rate, .max).
```xml
```
##### cloudwatch
Specifies that the metrics from this consumer should be forwarded to CloudWatch.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| region | required | string | |
Your AWS region
|
| namespace | required | string | |
The metrics namespace in CloudWatch
|
Example:
```xml
```
##### shared-credentials
Specifies that a profile from a shared-credentials file should be used for authentication to CloudWatch.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| file | required | string | |
The path to the shared-credentials file
|
| profile | optional | string | default |
The profile in the shared-credentials file
|
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [services.xml - 'admin'](#page-title)
- [adminserver](#adminserver)
- [cluster-controllers](#cluster-controllers)
- [cluster-controller](#cluster-controller)
- [configservers](#configservers)
- [configserver](#configserver)
- [logserver](#logserver)
- [slobroks](#slobroks)
- [slobrok](#slobrok)
- [monitoring](#monitoring)
- [logging](#logging)
- [metrics](#metrics)
- [consumer](#consumer)
- [metric-set](#metric-set)
- [metric](#metric)
- [cloudwatch](#cloudwatch)
- [shared-credentials](#shared-credentials)
---
### Application package reference
[](/en/reference/applications/application-packages.html.md "View as Markdown")
#### Application package reference
[](/en/reference/applications/application-packages.html.md "View as Markdown")
This is the [application package](../../basics/applications.html) reference. An application package is the deployment unit in Vespa. To deploy an application, create an application package and [vespa deploy](../../clients/vespa-cli.html#deployment) or use the [deploy API](../api/deploy-v2.html). The application package is a directory of files and subdirectories:
| Directory/file | Required | Description |
| --- | --- | --- |
| [services.xml](services/services.html) | Yes | Describes which services to run where, and their main configuration. |
| [hosts.xml](hosts.html) | No |
Vespa Cloud: Not used. See node counts in [services.xml](services/services.html).
Self-managed: The mapping from logical nodes to actual hosts.
|
| [deployment.xml](deployment.html) | Yes, for Vespa Cloud |
Specifies which environments and regions the application is deployed to during automated application deployment, as which application instances.
This file also specifies other deployment-related configurations like [cloud accounts](../../operations/enclave/enclave) and [private endpoints](../../operations/private-endpoints.html).
The file is required when deploying to the [prod environment](../../operations/environments.html#prod) - it is ignored (with some exceptions) when deploying to the _dev_ environment.
|
| [validation-overrides.xml](validation-overrides.html) | No | Override, allowing this package to deploy even if it fails validation. |
| [.vespaignore](../../applications/vespaignore.html) | No | Contains a list of path patterns that should be excluded from the `application.zip` deployed to Vespa. |
| [models](../ranking/model-files.html)/ | No | Machine-learned models in the application package. Refer to [stateless model evaluation](../../ranking/stateless-model-evaluation.html), [Tensorflow](../../ranking/tensorflow), [Onnx](../../ranking/onnx), [XGBoost](../../ranking/xgboost), and [LightGBM](../../ranking/lightgbm). |
| [schemas](../../basics/schemas.html)/ | No | Contains the \*.sd files describing the document types of the application and how they should be queried and processed. |
| [schemas/[schema]](../schemas/schemas.html#rank-profile)/ | No | Contains \*.profile files defining [rank profiles](../../basics/ranking.html#rank-profiles). This is an alternative to defining rank profiles inside the schema. |
| [security/clients.pem](../../security/guide) | Yes, for Vespa Cloud | PEM encoded X.509 certificates for data plane access. See the [security guide](../../security/guide) for how to generate and use. |
| [components](../../applications/components.html)/ | No | Contains \*.jar files containing searcher(s) for the JDisc Container. |
| [rules](../querying/semantic-rules.html)/ | No | Contains \*.sr files containing rule bases for semantic recognition and translation of the query |
| [search/query-profiles](../querying/query-profiles.html)/ | No | Contains \*.xml files containing a named set of search request parameters with values |
| [constants](../../ranking/tensor-user-guide.html#constant-tensors)/ | No | Constant tensors |
| [tests](testing.html)/ | No | Test files for automated tests |
| ext/ | No | Files that are guaranteed to be ignored by Vespa: They are excluded when processing the application package and cannot be referenced from any other element in it. |
Additional files and directories can be placed anywhere in the application package. These will be not be processed explicitly by Vespa when deploying the application package (i.e. they will only be considered if they are referred to from within the application package), but there is no guarantee to how these might be processed in a future release. To extend the application package in a way that is guaranteed to be ignored by Vespa in all future releases, use the _ext/_ directory.
##### Deploy
| Command | Description |
| --- | --- |
| upload | Uploads an application package to the config server. Normally not used, as _prepare_ includes _upload_ |
| prepare |
1. Verifies that a configuration server is up and running
2. Uploads the application to the configuration server, which stores it in _$VESPA\_HOME/var/db/vespa/config\_server/serverdb/tenants/default/sessions/[sessionid]_. _[sessionid]_ increases for each _prepare_-call. The config server also stores the application in a [ZooKeeper](/en/operations/self-managed/configuration-server.html) instance at _/config/v2/tenants/default/sessions/[sessionid]_ - this distributes the application to all config servers
3. Creates metadata about the deployed the applications package (which user deployed it, which directory was it deployed from and at what time was it deployed) and stores it in _...sessions/[sessionid]/.applicationMetaData_
4. Verifies that the application package contains the required files and performs a consistency check
5. Validates the xml config files using the [schema](https://github.com/vespa-engine/vespa/tree/master/config-model/src/main/resources/schema), found in _$VESPA\_HOME/share/vespa/schema_
6. Checks if there are config changes between the active application and this prepared application that require actions like restart or re-feed (like changes to [schemas](../../basics/schemas.html)). These actions are returned as part of the prepare step in the [deployment API](../api/deploy-v2.html#prepare-session). This prevents breaking changes to production - also read about [validation overrides](validation-overrides.html)
7. Distributes constant tensors and bundles with [components](../../applications/components.html) to nodes using [file distribution](/en/applications/deployment.html#file-distribution). Files are downloaded to _$VESPA\_HOME/var/db/vespa/filedistribution_, URL download starts downloading to _$VESPA\_HOME/var/db/vespa/download_
|
| activate |
1. Waits for prepare to complete
2. Activates new configuration version
3. Signals to containers to load new bundles - read more in [container components](../../applications/components.html)
|
| fetch | Use _fetch_ to download the active application package |
An application package can be zipped for deployment:
```
$ zip -r ../app.zip .
```
Use any name for the zip file - then refer to the file instead of the path in [deploy](../../clients/vespa-cli.html#deployment) commands.
**Important:** Using `tar` / `gzip` is not supported. [Details](https://github.com/vespa-engine/vespa/issues/17837).
##### Preprocess directives
Use preprocess directives to:
- _preprocess:properties_: define properties that one can refer to everywhere in _services.xml_
- _preprocess:include_: split _services.xml_ in smaller chunks
Below, _${container.port}_ is replaced by _4099_. The contents of _content.xml_ is placed at the _include_ point. This is applied recursively, one can use preprocess directives in included files, as long as namespaces are defined in the top level file:
```xml
4099
```
Sample _content.xml_:
```xml
1
```
##### Versioning application packages
An application can be given a user-defined version, available at [/ApplicationStatus](../../applications/components.html#monitoring-the-active-application). Configure the version in [services.xml](../applications/services/services.html) (at top level):
```xml
42
...
```
Copyright © 2026 - [Cookie Preferences](#)
---
### Vespa applications
[](/en/basics/applications.html.md "View as Markdown")
#### Vespa applications
[](/en/basics/applications.html.md "View as Markdown")
You use Vespa by deploying an _application_ to it. Why applications? Because Vespa handles both data and the computations you do over them - together an application.
An application is specified by an _application package_ - a directory with some files. The application package contains _everything_ that is needed to run your application: Config, schemas, components, ML models, and so on.
The _only_ way to change an application is to make the change in the application package and then deploy it again. Vespa will then safely change the running system to match the new application package revision, without impacting queries, writes, or data.
##### A minimal application package
You can create a complete application package with just a single file: services.xml. This file specifies the clusters that your application should run. It could just be a single stateless cluster - what's called _container_ - like this:
```xml
```
Put this in a file called services.xml, and you have created the world's smallest application package. However, this won't do much, usually you want to have a `content` cluster which can store data, maintain indexes, and run the distributed part of queries. You'll also want your container cluster to load the necessary middleware for this. With that we get a services file like this:
```xml
2
```
This specifies a pretty normal simple Vespa application, but now we need another file: The schema of the document type we'll use. This goes into the directory `schemas/`, so our application package now looks like this:
```
services.xml
schemas/myschema.sd
```
The schema file describes a kind of data and the computations (such as ranking/scoring) you want to do over it. At minimum it just lists the fields of that data type and if and each field should be indexed:
```vespa-schema-language
schema myschema {
document myschema {
field text type string {
indexing: summary | index
}
field embedding type tensor(x[384]) {
indexing: attribute | index
}
field popularity type double {
indexing: summary | attribute
}
}
}
```
With these two files we have specified a fully functional application that can do text, vector and hybrid search with filtering.
Rather than creating applications from scratch like this, you can also clone one of our sample applications as a starting point like we did in [getting started](deploy-an-application.html).
To read more on schemas, see the [schemas](schemas.html) guide. To see everything an application package can contain, see the [application package reference](../reference/applications/application-packages.html).
##### Deploying applications
To create running instances of an application, or make the changes to one take effect, you _deploy_ it. Deployments to the dev zone and to self-managed clusters sets up a single instance, while deployments to production can set up multiple instances in one or more regions.
To deploy an application package you use the [deploy command](../clients/vespa-cli.html#deployment) in Vespa CLI:
```shell
$ vespa deploy .
```
This will deploy the application package at the current dir to the current target and the default dev zone (use `vespa deploy -h` to see other options).
Deployment to production zones use a separate command:
```shell
$ vespa prod deploy .
```
Production deployments also require an additional file in the application package to specify where it should be deployed: deployment.xml. See [production deployment](../operations/production-deployment.html). The recommended way to deploy to production is by setting up a continuous deployment job, see [automated deployments](../operations/automated-deployments.html).
Deploying a change to an application package is generally safe to do at any time. It does not disrupt queries and writes, and invalid or destructive changes are rejected before taking effect. You can also add tests that verifies the application before deployment to production zones.
###### Next: [Schemas](schemas.html)
Copyright © 2026 - [Cookie Preferences](#)
---
### Component reference
[](/en/reference/applications/components.html.md "View as Markdown")
#### Component reference
[](/en/reference/applications/components.html.md "View as Markdown")
A component is any Java class whose lifetime is controlled by the container, see the [Developer Guide](../../applications/developer-guide.html) for an introduction. Components are specified and configured in services.xml and can have other components, and config (represented by generated "Config" classes) [injected](../../applications/dependency-injection.html) at construction time, and in turn be injected into other components.
Whenever a component or a resource your component depends on is changed by a redeployment, your component is reconstructed. Once all changed components are reconstructed, new requests are atomically switched to use the new set and the old ones are destructed.
If you have multiple constructors in your component, annotate the one to use for injection by `@com.yahoo.component.annotation.Inject`.
Identifiable components must implement `com.yahoo.component.Component`, and components that need to destruct resources at removal must subclass `com.yahoo.component.AbstractComponent` and implement `deconstruct()`.
See the [example](../../operations/metrics.html#example-qa) for common questions about component uniqueness / lifetime.
##### Component Types
Vespa defined various component types (superclasses) for common tasks:
| Component type | Description |
| --- | --- |
| Request handler |
[Request handlers](../../applications/request-handlers.html) allow applications to implement arbitrary HTTP APIs. A request handler accepts a request and returns a response. Custom request handlers are subclasses of [ThreadedHttpRequestHandler](https://javadoc.io/doc/com.yahoo.vespa/container-disc/latest/com/yahoo/container/jdisc/ThreadedHttpRequestHandler.html).
|
| Processor |
The [processing framework](../../applications/processing.html) can be used to create general composable synchronous request-response systems. Searchers and search chains are an instantiation (through subclasses) of this general framework for a specific domain. Processors are invoked synchronously and the response is a tree of arbitrary data elements. Custom output formats can be defined by adding [renderers](#renderers).
|
| Renderer |
Renderers convert a Response (or query Result) into a serialized form sent over the network. Renderers are subclasses of [com.yahoo.processing.rendering.Renderer](https://github.com/vespa-engine/vespa/blob/master/container-disc/src/main/java/com/yahoo/processing/rendering/Renderer.java).
|
| Searcher |
Searchers processes Queries and their Results. Since they are synchronous, they can issue multiple queries serially or in parallel to e.g. implement federation or decorate queries with information fetched from a content cluster. Searchers are composed into _search chains_ defined in services.xml. A query request selects a particular search chain which implements the logic of that query. [Read more](../../applications/searchers.html).
|
| Document processor |
Document processors processes incoming document operations. Similar to Searchers and Processors they can be composed in chains, but document processors are asynchronous. [Read more](../../applications/document-processors.html).
|
| Binding |
A binding matches a request URI to the correct [filter chain](#filter) or [request handler](#request-handlers), and route outgoing requests to the correct [client](#client). For instance, the binding _http://\*/\*_ would match any HTTP request, while _http://\*/processing_ would only match that specific path. If several bindings match, the most specific one is chosen.
| Server binding |
A server binding is a rule for matching incoming requests to the correct request handler, basically the JDisc building block for implementing RESTful APIs.
|
| Client binding |
A client binding is a pattern which is used to match requests originating inside the container, e.g. when doing federation, to a client provider. That is, it is a rule which determines what code should handle a given outgoing request.
|
|
| Filter |
A filter is a lightweight request checker. It may set some specific request property, or it may do security checking and simply block requests missing some mandatory property or header.
|
| Client |
Clients, or client providers, are implementations of clients for different protocols, or special rules for given protocols. When a JDisc application acts as a client, e.g. fetches a web page from another host, it is a client provider that handles the transaction. Bindings are used, as with request handlers and filters, to choose the correct client, matching protocol, server, etc., and then hands off the request to the client provider. There is no problem in using arbitrary other types of clients for external services in processors and request handlers.
|
##### Component configurations
This illustrates a typical component configuration set up by the Vespa container: 
The network layer associates a Request with a _response handler_ and routes it to the correct type of [request handler](#request-handlers) (typically based on URI binding patterns).
If an application needs lightweight request-response processing using decomposition by a series of chained logical units, the [processing framework](../../applications/processing.html) is the correct family of components to use. The request will be routed from ProcessingHandler through one or more chains of [Processor](#processors) instances. The exact format of the output is customizable using a [Renderer](#renderers).
If doing queries, SearchHandler will create a Query object, route that to the pertinent chain of [Searcher](#searchers) instances, and associate the returned Result with the correct [Renderer](#renderers) instance for optional customization of the output format.
The DocumentProcessingHandler is usually invoked from messagebus, and used for feeding documents into an index or storage. The incoming data is used to build a Document object, and this is then feed through a chain of [DocumentProcessor](#document-processors) instances.
If building an application with custom HTTP APIs, for instance arbitrary REST APIs, the easiest way is building a custom [RequestHandler](#request-handlers). This gets the Request, which is basically a set of key-value pairs, and returns a stream of arbitrary data back to the network.
##### Injectable Components
These components are available from Vespa for [injection](../../applications/dependency-injection.html) into applications in various contexts:
| Component | Description |
| --- | --- |
| Always available |
| --- |
| [AthenzIdentityProvider](https://github.com/vespa-engine/vespa/blob/master/container-disc/src/main/java/com/yahoo/container/jdisc/athenz/AthenzIdentityProvider.java) | Provides the application's Athenz-identity and gives access to identity/role certificate and tokens. |
| [BertBaseEmbedder](https://github.com/vespa-engine/vespa/blob/master/model-integration/src/main/java/ai/vespa/embedding/BertBaseEmbedder.java) | A BERT-Base compatible embedder, see [BertBase embedder](../../rag/embedding.html#bert-embedder). |
| [ConfigInstance](https://github.com/vespa-engine/vespa/blob/master/config-lib/src/main/java/com/yahoo/config/ConfigInstance.java) | Configuration is injected into components as `ConfigInstance` components - see [configuring components](../../applications/configuring-components.html). |
| [Executor](https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Executor.html) | Default threadpool for processing requests in threaded request handler |
| [Linguistics](https://github.com/vespa-engine/vespa/blob/master/linguistics/src/main/java/com/yahoo/language/Linguistics.java) | Inject a Linguistics component like [SimpleLinguistics](https://github.com/vespa-engine/vespa/blob/master/linguistics/src/main/java/com/yahoo/language/simple/SimpleLinguistics.java) or provide a custom implementation - see [linguistics](../../linguistics/linguistics.html). |
| [Metric](https://github.com/vespa-engine/vespa/blob/master/jdisc_core/src/main/java/com/yahoo/jdisc/Metric.java) | Jdisc core interface for metrics. Required by all subclasses of ThreadedRequestHandler. |
| [MetricReceiver](https://github.com/vespa-engine/vespa/blob/master/container-disc/src/main/java/com/yahoo/metrics/simple/MetricReceiver.java) | Use to emit metrics from a component. Find an example in the [metrics](../../operations/metrics.html#metrics-from-custom-components) guide. |
| [ModelsEvaluator](https://github.com/vespa-engine/vespa/blob/master/model-evaluation/src/main/java/ai/vespa/models/evaluation/ModelsEvaluator.java) | Evaluates machine-learned models added to Vespa applications and available as config form. |
| [SentencePieceEmbedder](https://github.com/vespa-engine/vespa/blob/master/linguistics-components/src/main/java/com/yahoo/language/sentencepiece/SentencePieceEmbedder.java) | A native Java implementation of SentencePiece, see [SentencePiece embedder](../rag/embedding.html#sentencepiece-embedder). |
| [VespaCurator](https://github.com/vespa-engine/vespa/blob/master/zkfacade/src/main/java/com/yahoo/vespa/curator/api/VespaCurator.java) |
A client for ZooKeeper. For use in container clusters that have ZooKeeper enabled. See [using ZooKeeper](../../applications/using-zookeeper).
|
| [VipStatus](https://github.com/vespa-engine/vespa/blob/master/container-disc/src/main/java/com/yahoo/container/handler/VipStatus.java) | Use this to gain control over the service status (up/down) to be emitted from this container. |
| [WordPieceEmbedder](https://github.com/vespa-engine/vespa/blob/master/linguistics-components/src/main/java/com/yahoo/language/wordpiece/WordPieceEmbedder.java) | An implementation of the WordPiece embedder, usually used with BERT models. Refer to [WordPiece embedder](../rag/embedding.html#wordpiece-embedder). |
| [SystemInfo](https://github.com/vespa-engine/vespa/blob/master/hosted-zone-api/src/main/java/ai/vespa/cloud/SystemInfo.java) | Vespa Cloud: Provides information about the environment the component is running in. [Read more](/en/applications/components.html#the-systeminfo-injectable-component). |
| Available in containers having `search` |
| --- |
| [DocumentAccess](https://github.com/vespa-engine/vespa/blob/master/documentapi/src/main/java/com/yahoo/documentapi/DocumentAccess.java) | To use the [Document API](../../writing/document-api-guide.html). |
| [ExecutionFactory](https://github.com/vespa-engine/vespa/blob/master/container-search/src/main/java/com/yahoo/search/searchchain/ExecutionFactory.java) | To execute new queries from code. [Read more](../../applications/web-services.html#queries). |
| [Map\](https://github.com/vespa-engine/vespa/blob/master/model-evaluation/src/main/java/ai/vespa/models/evaluation/Model.java) | Use to inject a set of Models, see [Stateless Model Evaluation](../../ranking/stateless-model-evaluation.html). |
| Available in containers having `document-api` or `document-processing` |
| --- |
| [DocumentAccess](https://github.com/vespa-engine/vespa/blob/master/documentapi/src/main/java/com/yahoo/documentapi/DocumentAccess.java) | To use the [Document API](../../writing/document-api-guide.html). |
##### Component Versioning
Components as well as many other artifacts in the container can be versioned. This document explains the format and semantics of these versions and how they are referred.
###### Format
Versions are on the form:
```
version ::= major ["." minor [ "." micro [ "." qualifier]]]
```
Where `major`, `minor`, and `micro` are integers and `qualifier` is any string.
A version is appended to an id separated by a colon. In cases where a file is created for each component version, the colon is replaced by a dash in the file name.
###### Ordering
Versions are ordered first by major, then minor, then micro and then by doing a lexical ordering on the qualifier. This means that `a:1 < a:1.0 < a:1.0.0 < a:1.1 < a:1.1.0 < a:2`
###### Referencing a versioned Component
Whenever component is referenced by id (in code or configuration), a fully or partially specified version may be included in the reference by using the form `id:versionSpecification`. Such references are resolved using the following rules:
- An id without any version specification resolves to the highest version not having a qualifier.
- A partially or full version specification resolves to the highest version not having a qualifier which matches the specification.
- Versions with qualifiers are matched only by exact match.
Example: Given a component with id `a` having these versions: `[1.1, 1.2, 1.2, 1.3.test, 2.0]`
- The reference `a` will resolve to `a:2.0`
- The reference `a:1` will resolve to `a:1.2`
- The only way to resolve to the "test" qualified version is by using the exact reference `a:1.3.test`
- These references will not resolve: `a:1.3`, `a:3`, `1.2.3`
###### Merging specifications for chained Components
In some cases, there is a need for merging multiple references into one. An example is inheritance of chains of version references, where multiple inherited chains may reference the same component.
Two version references are said to be _compatible_ if one is a prefix of the other. In this case the most specific version is used. If they are not compatible they are _conflicting_. Example:
```xml
bundle="the name in in your pom.xml"
bundle="the name in in your pom.xml"
bundle="the name in in your pom.xml"
```
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Component reference](#page-title)
- [Component Types](#component-types)
- [Component configurations](#component-configurations)
- [Injectable Components](#injectable-components)
- [Component Versioning](#component-versioning)
- [Format](#format)
- [Ordering](#ordering)
- [Referencing a versioned Component](#referencing-a-versioned-component)
- [Merging specifications for chained Components](#merging-specifications-for-chained-components)
---
### Custom Configuration File Reference
[](/en/reference/applications/config-files.html.md "View as Markdown")
#### Custom Configuration File Reference
[](/en/reference/applications/config-files.html.md "View as Markdown")
This is the reference for config file definitions. It is useful for developing applications that has [configurable components](../../applications/configuring-components.html) for the [Vespa Container](../../applications/containers.html), where configuration for individual components may be provided by defining [``](#generic-configuration-in-services-xml) elements within the component's scope in services.xml.
##### Config definition files
Config definition files are part of the source code of your application and have a _.def_ suffix. Each file defines and documents the content and semantics of one configuration type. Vespa's builtin _.def_ files are found in `$VESPA_HOME/share/vespa/configdefinitions/`.
###### Package
Package is a mandatory statement that is used to define the package for the java class generated to represent the file. For [container component](../../applications/components.html) developers, it is recommended to use a separate package for each bundle that needs to export config classes, to avoid conflicts between bundles that contain configurable components. Package must be the first non-comment line, and can only contain lower-case characters and dots:
```
package=com.mydomain.mypackage
```
###### Parameter names
Config definition files contain lines on the form:
```
parameterName type [default=value] [range=[min,max]]
```
camelCase in parameter names is recommended for readability.
###### Parameter types
Supported types for variables in the _.def_ file:
| int |
32 bit signed integer value
|
| long |
64 bit signed integer value
|
| double |
64 bit IEEE float value
|
| enum |
Enumerated types. A set of strings representing the valid values for the parameter, e.g:
```
foo enum {BAR, BAZ, QUUX} default=BAR
```
|
| bool |
A boolean (true/false) value
|
| string |
A String value. Default values must be enclosed in quotation marks (" "), and any internal quotation marks must be escaped by backslash. Likewise, newlines must be escaped to `\n`
|
| path |
A path to a physical file or directory in the application package. This makes it possible to access files from the application package in container components. The path is relative to the root of the [application package](../../basics/applications.html). A path parameter cannot have a default value, but may be optional (using the _optional_ keyword after the type). An optional path does not have to be set, in which case it will be an empty value. The content will be available as a `java.nio.file.Path` instance when the component accessing this config is constructed, or an `Optional` if the _optional_ keyword is used.
|
| url |
Similar to `path`, an arbitrary URL of a file that should be downloaded and made available to container components. The file content will be available as a java.io.File instance when the component accessing this config is constructed. Note that if the file takes a long time to download, it will also take a long time for the container to come up with the configuration referencing it. See also the [note about changing contents for such a url](../../applications/configuring-components.html#adding-files-to-the-component-configuration).
|
| model |
A pointer to a machine-learned model. This can be a model-id, url or path, and multiple of these can be specified as a single config value, where one is used depending on the deployment environment:
- If a model-id is specified and the application is deployed on Vespa Cloud, the model-id is used.
- Otherwise, if a URL is specified, it is used.
- Otherwise, path is used.
You may also use remote URLs protected by bearer-token authentication by supplying the optional `secret-ref` attribute. See [using private Huggingface models](../rag/embedding.html#private-model-hub).
On the receiving side, this config value is simply represented as a file path regardless of how it is resolved. This makes it easy to refer to models in multiple ways such that the appropriate one is used depending on the context. The special syntax for setting these config values is documented in [adding files to the configuration](../../applications/configuring-components.html#adding-files-to-the-component-configuration). |
| reference |
A config id to another configuration (only for internal vespa usage)
|
###### Structs
Structs are used to group a number of parameters that naturally belong together. A struct is declared by adding a '.' between the struct name and each member's name:
```
basicStruct.foo string
basicStruct.bar int
```
###### Arrays
Arrays are declared by appending square brackets to the parameter name. Arrays can either contain simple values, or have children. Children can be simple parameters and/or structs and/or other arrays. Arbitrarily complex structures can be built to any depth. Examples:
```
intArr[] int # Integer value array
row[].column[] int # Array of integer value arrays
complexArr[].foo string # Complex array that contains
complexArr[].bar double # … two simple parameters
complexArr[].coord.x int # … and a struct called 'coord'
complexArr[].coord.y int
complexArr[].coord.depths[] double # … that contains a double array
```
Note that arrays cannot have default values, even for simple value arrays. An array that has children cannot contain simple values, and vice versa. In the example above, `intArr` and `row.column` could not have children, while `row` and `complexArr` are not allowed to contain values.
###### Maps
Maps are declared by appending curly brackets to the parameter name. Arbitrarily complex structures are supported also here. Examples:
```
myMap{} int
complexMap{}.nestedMap{}.id int
complexMap{}.nestedMap{}.name string
```
##### Generic configuration in services.xml
`services.xml`has four types of elements:
| individual service elements | (e.g. _searcher_, _handler_, _searchnode_) - creates a service, but has no child elements that create services |
| service group elements | (e.g. _content_, _container_, _document-processing_ - creates a group of services and can have all types of child elements |
| dedicated config elements | (e.g. _accesslog_) - configures a service or a group of services and can only have other dedicated config elements as children |
| generic config elements | always named _config_ |
Generic config elements can be added to most elements that lead to one or more services being created - i.e. service group elements and individual service elements. The config is then applied to all services created by that element and all descendant elements.
For example, by adding _config_ for _container_, the config will be applied to all container components in that cluster. Config at a deeper level has priority, so this config can be overridden for individual components by setting the same config values in e.g. _handler_ or _server_ elements.
Given the following config definition, let's say its name is `type-examples.def`:
```
package=com.mydomain
stringVal string
myArray[].name string
myArray[].type enum {T1, T2, T3} default=T1
myArray[].intArr[] int
myMap{} string
basicStruct.foo string
basicStruct.bar int default=0 range=[-100,100]
boolVal bool
myFile path
myUrl url
myOptionalPath path optional
```
To set all the values for this config in `services.xml`, add the following xml at the desired element (the name should be _\.\_):
```xml
val
-
elem_0
T2
- 0
- 1
-
elem_1
T3
- 0
- 1
- val1
- val2
str
3
true
components/file1.txt
https://docs.vespa.ai/en/reference/query-api-reference.html
```
Note that each '.' in the parameter's definition corresponds to a child element in the xml. It is not necessary to set values that already have a default in the _.def_ file, if you want to keep the default value. Hence, in the example above, `basicStruct.bar` and `myArray[].type` could have been omitted in the xml without generating any errors when deploying the application.
###### Configuring arrays
Assigning values to _arrays_ is done by using the `- ` element. This ensures that the given config values do not overwrite any existing array elements from higher-level xml elements in services, or from Vespa itself.
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Custom Configuration File Reference](#page-title)
- [Config definition files](#config-definition-files)
- [Package](#package)
- [Parameter names](#parameter-names)
- [Parameter types](#parameter-types)
- [Structs](#structs)
- [Arrays](#arrays)
- [Maps](#maps)
- [Generic configuration in services.xml](#generic-configuration-in-services-xml)
- [Configuring arrays](#configuring-arrays)
---
### services.xml - container
[](/en/reference/applications/services/container.html.md "View as Markdown")
#### services.xml - container
[](/en/reference/applications/services/container.html.md "View as Markdown")
This is the reference for the container cluster configuration in [services.xml](services.html).
```
container [version, id][http](http.html)[server [id, port]](http.html#server)[filtering](http.html#filtering)[handler [id, class, bundle]](#handler)[binding](#binding)[component](#component)[server [id, class, bundle]](#server)[clients](#clients)[client [id, permissions]](#client)[certificate [file]](#certificate)[token [id]](#token)[components](#components)[component](#component)[search](search.html)[include [dir]](#include)[binding](search.html#binding)[searcher](search.html#searcher)[federation](search.html#federation)[provider](search.html#provider)[chain](processing.html#chain)[renderer](search.html#renderer)[threadpool](search.html#threadpool)[significance](search.html#significance)[skills](#skills)[schemas](#schemas)[document-processing](docproc.html)[include [dir]](#include)[documentprocessor](docproc.html#documentprocessor)[chain](processing.html#chain)[threadpool](docproc.html#threadpool)[processing](processing.html)[include [dir]](#include)[binding](processing.html#binding)[processor](processing.html#processor)[chain](processing.html#chain)[renderer](processing.html#renderer)[document-api](#document-api)[abortondocumenterror](#abortondocumenterror)[ignore-undefined-fields](#ignore-undefined-fields)[legacy-feed-api](#legacy-feed-api)[maxpendingdocs](#maxpendingdocs)[maxpendingbytes](#maxpendingbytes)[mbusport](#mbusport)[max-document-size](#max-document-size)[retrydelay](#retrydelay)[retryenabled](#retryenabled)[route](#route)[timeout](#timeout)[tracelevel](#tracelevel)[model-evaluation](../../../ranking/stateless-model-evaluation.html)[onnx](../../../ranking/stateless-model-evaluation.html#onnx-inference-options)[inference](#inference)[memory](#inference-memory)[document [type, class, bundle]](#document)[accesslog [type, fileNamePattern, symlinkName, rotationInterval, rotationScheme]](#accesslog)[request-content [samples-per-second, path-prefix, max-bytes]](#request-content)[config](../config-files.html#generic-configuration-in-services-xml)[nodes [count, allocated-memory, jvm-gc-options, jvm-options]](#nodes)[resources](services.html#resources)(Vespa Cloud)[node [hostalias]](services.html#node)(Self managed)[environment-variables](#environment-variables)[jvm [allocated-memory, options, gc-options]](#jvm)[secrets](#secrets)[secret-store [type]](#secret-store)[group [name, environment]](#group)[zookeeper](#zookeeper)[threadpool](#threadpool)
```
[config](../config-files.html#generic-configuration-in-services-xml) elements are also allowed most places.
Example:
```
```
##### \
Contained in [\](services.html). Each container tag specifies a separate container cluster.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| version | required | number | |
1.0 in this version of Vespa
|
| id | required | string | |
the id of this cluster
|
##### handler
The `handler` element holds the configuration of a request handler. For each `binding` tag, the handler will be bound to the pertinent JDisc interfaces using the given binding.
- `binding` For JDisc request handlers, add this server binding to this handler.
- [`component`](#component) for injecting another component. Must be a declaration of a new component, not a reference.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The component ID
|
| class | optional | string | |
The class of the handler, defaults to id
|
| bundle | optional | string | |
The bundle to load the handler from: The name in \ in pom.xml. Defaults to class or id (if no class is given)
|
Example:
```xml
http://*/*
```
##### binding
The URI to map a Handler to. Multiple elements are allowed. See example above.
##### server
The `server` element holds the configuration of a JDisc server provider.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The component ID
|
| class | optional | string | |
The class of the server, defaults to id
|
| bundle | optional | string | |
The bundle to load the server from: The name in \ in the pom.xml. Defaults to class or id (if no class is given).
|
Example:
```xml
Hello, world!
16889
```
##### clients
Vespa Cloud only. The `clients` element is a parent element for [client](#client) security configuration. Find details and practical examples in the [security guide](../../../security/guide#configuring-tokens). Example:
```xml
```
##### client
Vespa Cloud only. Child element of [clients](#clients). Use to configure security credentials for a container cluster, using [certificate](#certificate) and/or [token](#token).
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The client ID
|
| permissions | required | string | |
Permissions, see the [security guide](../../../security/guide#permissions). One of:
- `read`
- `write`
- `read,write`
|
##### certificate
Vespa Cloud only. Child element of [client](#client). Configure certificates using the _file_ attribute.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| file | required | string | |
Path to the certificate file, see the [security guide](../../../security/guide#configuring-mtls).
|
##### token
Vespa Cloud only. Child element of [client](#client). Configure tokens using the _id_ attribute.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
Token ID, see the [security guide](../../../security/guide#configuring-tokens).
|
##### components
Contains [component](#component) elements. Can be used in conjunction with [include](#include) for modular config of components.
##### component
The `component` element holds the configuration of a [generic component](../../../applications/dependency-injection.html#special-components). Must be a declaration of a new component, not a reference.
Nested [`component`](#component) child elements can be added for injecting specific component instances. This is useful if there is more than one declared component of the same Java class. Refer to [Injecting components](../../../applications/dependency-injection.html) for details and examples.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The component ID
|
| class | optional | string | |
The class of the component, defaults to id
|
| bundle | optional | string | |
The bundle to load the component from: The name in \ in the pom.xml. Defaults to class or id (if no class is given).
|
Example:
```xml
```
##### skills
Adds the `skills/v1` API to this container to provide agent users with information about how to use this service. This may contain [schemas](#schemas) element.
##### schemas
Adds the `schemas` skill to this container at the path `skills/v1/schemas`. This provides agent users with information on how to query the data in this application, see [agent-ready schemas](../../../rag/agent-ready-schemas).
.
This requires the `search` tag to be present.
##### document-api
Use to enable [Document API](../../api/api.html) operations to a container cluster. Child elements:
| Name | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| abortondocumenterror | optional | true/false | true |
Controls whether to abort the entire feed or not if a document-related error occurs, i.e. if a document contains an unknown field. Setting this field to `true` will abort the feed on such errors, while setting it to `false` will cause Vespa to simply skip to the next document in the feed. Note that malformed XML in the input will abort the feed regardless of this setting.
|
| binding | optional | string | http://\*/ |
The URI to map the document-api handlers to. Multiple bindings are valid. Must end with a '/'. Note that each document-api handler will get its individual binding by adding a suffix, e.g. the feed handler will add 'feed/', the remove handler will add 'remove/' and so on. Example:
```xml
http://*/document-api/
https://*/document-api/
```
With these configured bindings, the feed handler will be available at `http://*/document-api/feed/` and `https://*/document-api/feed/`. For other handlers, just replace 'feed/' with the appropriate suffix, e.g. 'get/', 'remove/' etc. |
| ignore-undefined-fields | optional | true/false | false |
Set to true to ignore undefined fields in document API operations and let such operations complete successfully, rather than fail. A [response header is returned](../../api/document-v1.html#x-vespa-ignored-fields) when field operations are ignored.
|
| legacy-feed-api | optional | disabled/enabled | enabled |
Set to disabled to not set up legacy feed API (which is only used by the legacy [vespa-http-client](https://docs.vespa.ai/en/reference/release-notes/vespa8.html#vespa-http-client).
|
| max-document-size | optional | string | 100MiB |
Specifies the maximum size of a document operation request accepted by the container, measured as the uncompressed size of the request body. The limit applies to all document types in the container cluster. A request larger than this limit will be rejected by the container before the operation is forwarded to the content cluster.
Valid values are numbers including a unit (e.g. _10MiB_) and the value must be between 1MiB and 2048MiB (inclusive). Values will be rounded to the nearest MiB, so using MiB as a unit is preferable.
The value should normally not exceed the smallest [max-document-size](content.html#max-document-size) configured in any content cluster that this container feeds to; a deployment warning is emitted otherwise.
Example:
```xml
10MiB
```
|
| maxpendingbytes | optional | number | |
The maximum number of pending bytes. If `` is 0 and this is set to 0, this defaults to 100 MB. If `` is more than 0, and this is set to 0, the send-window is only limited by number of messages sent, not the memory footprint.
|
| maxpendingdocs | optional | number | |
The maximum number of pending documents the client can have. By default, the client will dynamically adjust the window size based on the latency of the performed operations. If the parameter is set, dynamic window sizing will be turned off in favor of the configured value.
|
| mbusport | optional | number | |
Set the MessageBus port
|
| retrydelay | optional | double | 1.0 |
Delay in seconds between retries
|
| retryenabled | optional | true/false | |
Enable or disable retrying documents that have failed.
|
| route | optional | string | default |
Set the route to feed documents to
|
| timeout | optional | double | 180.0 |
Set the timeout value in seconds for an operation
|
| tracelevel | optional | 0-9 | 0 |
Configure the level of which to trace messages sent. The higher the level, the more detailed descriptions.
|
Example:
```xml
http://*/document-api/
https://*/document-api/
false
1048576
1000
1234
5.5
false
default
250.5
3
10MiB
```
##### inference
Vespa Cloud only. Configures resources used for model inference in the container, for example [embedders](../../../rag/embedding.html), [local LLMs](../../../rag/local-llms.html), and [stateless model evaluation](../../../ranking/stateless-model-evaluation.html).
| Element | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| memory | optional | string | auto-estimated |
Container memory reserved for model inference, covering both model weights and inference requests. This memory is subtracted from the memory available to the JVM heap on the same node.
When not set, Vespa estimates the required inference memory automatically. The automatic estimate can be inaccurate for some models and workloads, which may lead to out-of-memory errors. Set this element explicitly to override the estimate.
|
Example:
```xml
1500Mb
...
```
##### document
[Concrete document type](../../../schemas/concrete-documents.html) bindings for the container. Example:
```xml
...
```
##### accesslog
Configures properties of the accesslog. The default type is `json` that will give output in (line-based) [JSON format](../../../operations/access-logging.html). See [Access logging](../../../operations/access-logging.html) for configuration details. Setting the type to `vespa` gives a classic Apache CLF-like format.
Access logging can be disabled by setting the type to `disabled`.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| type | optional | string | json |
The accesslog type: _json_, _vespa_ or _disabled_
|
| fileNamePattern | required\* | string | JsonAccessLog.\.%Y%m%d%H%M%S |
File name pattern. \* Note: Optional when _type_ is _disabled_
|
| symlinkName | optional | string | JsonAccessLog.\ |
Symlink name
|
| rotationInterval | optional | string | 0 60 ... |
Rotation interval
|
| rotationScheme | optional | string | date |
Valid values are _date_ or _sequence_
|
###### request-content
The `request-content` element is a child of `accesslog` and configures logging of request content. Multiple `request-content` elements can be specified to log different request paths with different configurations.
| Element | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| samples-per-second | required | double | |
Probabilistic sample rate per second
|
| path-prefix | required | string | |
URI path prefix to match for logging
|
| max-bytes | required | integer | |
Maximum size in bytes to log, only prefix will be kept for larger requests
|
Example:
```xml
0.2
/search
65536
```
##### include
Allows including XML snippets contained in external files. All files from all listed directories will be included. All files must have the same outer tag as they were referred from, i.e. search, document-processing or processing. The path must be relative to the application package root, and must never point outside the package.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| dir | required | string | |
The directory to include files from. File inclusion order is undefined.
|
Example:
```xml
```
##### nodes
See [nodes](services.html#nodes) in the general services.xml documentation.
Additional container cluster specific attributes:
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| allocated-memory | optional | percentage | |
**Deprecated:** See [jvm](#jvm).
|
| jvm-options | optional | string | |
**Deprecated:** See [jvm](#jvm).
|
| jvm-gc-options | optional | string | |
**Deprecated:** See [jvm](#jvm).
|
##### environment-variables
Add child elements to set environment variables accessible in the container JVM runtime.
Example:
```xml
1
granularity=fine,verbose,compact,1,0
```
##### jvm
JVM settings for container nodes.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| allocated-memory | optional | percentage | 85% | Memory to allocate to each JVM instance as a percentage of available memory. Must be an integer percentage followed by _%_. If not set, and running self-hosted, the heap size is set to 1.5G. |
| options | optional | string | | Generic JVM options |
| gc-options | optional | string | | JVM GC options. Garbage Collector specific parameters |
Example where 50% of the node total memory is used as the Max heap size of the JVM:
```xml
```
##### secrets
Use to access secrets configured in Vespa Cloud - refer to the [secret store](../../../security/secret-store).
##### secret-store
The `secret-store` element holds configuration for custom implementations. Contains one or more `group` elements.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| type | required | string | |
Value: "oath-ckms"
|
##### group
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| name | required | string | |
Key group name
|
| environment | required | string | |
Value one of: "alpha" "corp" "prod" "aws" "aws\_stage"
|
Example:
```xml
```
##### zookeeper
The _zookeeper_ element declares that the container cluster should run ZooKeeper and configure the necessary components. This element has no attributes or children.
##### threadpool
Available since Vespa 8.611.13 .
Specifies configuration for the default thread pool in the container. All parameters are relative to the number of CPU cores—see the [container tuning example](../../../performance/container-tuning.html#container-worker-threads-example). This thread pool also supports the optional `max` attribute, which lets the pool grow up to `max * vCPU` threads under load before shrinking back after 5 seconds of idleness. Requests are rejected once the allowed number of threads is reached, all are busy, and the queue is full.
###### threads
The number of permanent threads relative to number of vCPU cores. Default value is `2`.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| max | optional | number | 100 |
The maximum number of threads relative to vCPU cores. Value must be greater than or equal to ``.
|
###### queue
The size of the request queue relative to effective number of threads. Specify `0` to disable queuing. Queueing is disabled by default.
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [services.xml - container](#page-title)
- [\](#)
- [handler](#handler)
- [binding](#binding)
- [server](#server)
- [clients](#clients)
- [client](#client)
- [certificate](#certificate)
- [token](#token)
- [components](#components)
- [component](#component)
- [skills](#skills)
- [schemas](#schemas)
- [document-api](#document-api)
- [inference](#inference)
- [document](#document)
- [accesslog](#accesslog)
- [request-content](#request-content)
- [include](#include)
- [nodes](#nodes)
- [environment-variables](#environment-variables)
- [jvm](#jvm)
- [secrets](#secrets)
- [secret-store](#secret-store)
- [group](#group)
- [zookeeper](#zookeeper)
- [threadpool](#threadpool)
- [threads](#threadpool-threads)
- [queue](#threadpool-queue)
---
### services.xml - 'content'
[](/en/reference/applications/services/content.html.md "View as Markdown")
#### services.xml - 'content'
[](/en/reference/applications/services/content.html.md "View as Markdown")
```
[content](#content)[documents](#documents)[document](#document)[document-processing](#document-processing)[min-redundancy](#min-redundancy)[redundancy](#redundancy)[coverage-policy](#coverage-policy)[nodes](services.html#nodes)[node](#node)[group](#group)[distribution](#distribution)[node](#node)[group](#group)[engine](#engine)[proton](#proton)[searchable-copies](#searchable-copies)[tuning](#tuning-proton)[searchnode](#searchnode)[lidspace](#lidspace)[max-bloat-factor](#lidspace-max-bloat-factor)[requestthreads](#requestthreads)[search](#requestthreads-search)[persearch](#requestthreads-persearch)[summary](#requestthreads-summary)[resultprocessing](#requestthreads-resultprocessing)[flushstrategy](#flushstrategy)[native](#flushstrategy-native)[total](#flushstrategy-native-total)[maxmemorygain](#flushstrategy-native-total-maxmemorygain)[diskbloatfactor](#flushstrategy-native-total-diskbloatfactor)[component](#flushstrategy-native-component)[maxmemorygain](#flushstrategy-native-component-maxmemorygain)[diskbloatfactor](#flushstrategy-native-component-diskbloatfactor)[maxage](#flushstrategy-native-component-maxage)[transactionlog](#flushstrategy-native-transactionlog)[maxsize](#flushstrategy-native-transactionlog-maxsize)[conservative](#flushstrategy-native-conservative)[memory-limit-factor](#flushstrategy-native-conservative-memory-limit-factor)[disk-limit-factor](#flushstrategy-native-conservative-disk-limit-factor)[initialize](#initialize)[threads](#initialize-threads)[feeding](#feeding)[concurrency](#feeding-concurrency)[niceness](#feeding-niceness)[index](#index)[io](#index-io)[search](#index-io-search)[warmup](#index-warmup)[time](#index-warmup-time)[unpack](#index-warmup-unpack)[removed-db](#removed-db)[prune](#removed-db-prune)[age](#removed-db-prune-age)[interval](#removed-db-prune-interval)[summary](#summary)[io](#summary-io)[read](#summary-io-read)[store](#summary-store)[cache](#summary-store-cache)[maxsize](#summary-store-cache-maxsize)[maxsize-percent](#summary-store-cache-maxsize-percent)[compression](#summary-store-cache-compression)[type](#summary-store-cache-compression-type)[level](#summary-store-cache-compression-level)[logstore](#summary-store-logstore)[maxfilesize](#summary-store-logstore-maxfilesize)[chunk](#summary-store-logstore-chunk)[maxsize](#summary-store-logstore-chunk-maxsize)[compression](#summary-store-logstore-chunk-compression)[type](#summary-store-logstore-chunk-compression-type)[level](#summary-store-logstore-chunk-compression-level)[sync-transactionlog](#sync-transactionlog)[flush-on-shutdown](#flush-on-shutdown)[resource-limits](#resource-limits-proton)[disk](#disk)[memory](#memory)[search](#search)[query-timeout](#query-timeout)[visibility-delay](#visibility-delay)[coverage](#coverage)[minimum](#minimum)[min-wait-after-coverage-factor](#min-wait-after-coverage-factor)[max-wait-after-coverage-factor](#max-wait-after-coverage-factor)[tuning](#tuning)[bucket-splitting](#bucket-splitting)[min-node-ratio-per-group](#min-node-ratio-per-group)[distribution](#distribution_type)[max-document-size](#max-document-size)[merges](#merges)[persistence-threads](#persistence-threads)[resource-limits](#resource-limits)[visitors](#visitors)[max-concurrent](#max-concurrent)[dispatch](#dispatch-tuning)[max-hits-per-partition](#max-hits-per-partition)[dispatch-policy](#dispatch-policy)[prioritize-availability](#prioritize-availability)[min-active-docs-coverage](#min-active-docs-coverage)[top-k-probability](#top-k-probability)[cluster-controller](#cluster-controller)[init-progress-time](#init-progress-time)[transition-time](#transition-time)[max-premature-crashes](#max-premature-crashes)[stable-state-period](#stable-state-period)[min-distributor-up-ratio](#min-distributor-up-ratio)[min-storage-up-ratio](#min-storage-up-ratio)[groups-allowed-down-ratio](#groups-allowed-down-ratio)
```
##### content
The root element of a Content cluster definition. Creates a content cluster. A content cluster stores and/or indexes documents. The xml file may have zero or more such tags.
Contained in [services](services.html).
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| version | required | number | | 1.0 in this version of Vespa |
| id | required for multiple clusters | string | |
Name of the content cluster. If none is supplied, the cluster name will be `content`. Cluster names must be unique within the application, if multiple clusters are configured, the name must be set for all but one at minimum.
**Note:** Renaming a cluster is the same as dropping the current cluster and adding a new one. This makes data unavailable or lost, depending on hosting model. Deploying with a changed cluster id will therefore fail with a validation override requirement: `Content cluster 'music' is removed. This will cause loss of all data in this cluster.
To allow this add content-cluster-removal to validation-overrides.xml,
see https://docs.vespa.ai/en/reference/validation-overrides.html`.
|
Subelements:
- [documents](#documents) (required)
- [min-redundancy](#min-redundancy)
- [redundancy](#redundancy)
- [coverage-policy](#coverage-policy)
- [nodes](services.html#nodes)
- [group](#group)
- [engine](#engine)
- [search](#search)
- [tuning](#tuning)
##### documents
Contained in [content](#content). Defines which document types should be routed to this content cluster using the default route, and what documents should be kept in the cluster if the garbage collector runs. Read more on [expiring documents](../../../schemas/documents.html#document-expiry). Also have some backend specific configuration for whether documents should be searchable or not.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| selection | optional | string | |
A [document selection](../../writing/document-selector-language.html), restricting documents that are routed to this cluster. Defaults to a selection expression matching everything.
This selection can be specified to match document identifier specifics that are _independent_ of document types. For restrictions that apply only to a _specific_ document type, this must be done within that particular document type's [document](#document) element. Trying to use document type references in this selection makes an error during deployment. The selection given here will be merged with per-document type selections specified within document tags, if any, meaning that any document in the cluster must match _both_ selections to be accepted and kept.
This feature is primarily used to [expire documents](../../../schemas/documents.html#document-expiry).
|
| garbage-collection | optional | true / false | false |
If true, regularly verify the documents stored in the cluster to see if they belong in the cluster, and delete them if not. If false, garbage collection is not run.
|
| garbage-collection-interval | optional | integer | 3600 |
Time (in seconds) between garbage collection cycles. Note that the deletion of documents is spread over this interval, so more resources will be used for deleting a set of documents with a small interval than with a larger interval.
|
Subelements:
- [document](#document) (required)
- [document-processing](#document-processing) (optional)
##### document
Contained in [documents](#documents). The document type to be routed to this content cluster.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| type | required | string | |
[Document type name](../../schemas/schemas.html#document)
|
| mode | required | index /
store-only /
streaming | |
The mode of storing and indexing. Refer to [streaming search](../../../performance/streaming-search.html) for _store-only_, as documents are stored the same way for both cases.
Changing mode requires an _indexing-mode-change_[validation override](../validation-overrides.html), and documents must be re-fed.
|
| selection | optional | string | |
A [document selection](../../writing/document-selector-language.html), restricting documents that are routed to this cluster. Defaults to a selection expression matching everything.
This selection must apply to fields in _this document type only_. Selection will be merged together with selection for other types and global selection from [documents](#documents) to form a full expression for what documents belong to this cluster.
|
| global | optional | true / false | false |
Set to _true_ to distribute all documents of this type to all nodes in the content cluster it is defined.
Fields in global documents can be imported into documents to implement joins - read more in [parent/child](../../../schemas/parent-child.html). Vespa will detect when a new (or outdated) node is added to the cluster and prevent it from taking part in searches until it has received all global documents.
Changing from _false_ to _true_ or vice versa requires a _global-document-change_[validation override](../validation-overrides.html). First, [stop services](/en/operations/self-managed/admin-procedures.html#vespa-start-stop-restart) on all content nodes. Then, deploy with the validation override. Finally, [start services](/en/operations/self-managed/admin-procedures.html#vespa-start-stop-restart) on all content nodes.
Note: _global_ is only supported for _mode="index"_.
|
##### document-processing
Contained in [documents](#documents). Vespa Search specific configuration for which document processing cluster and chain to run index preprocessing.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| cluster | optional | string | Container cluster on content node |
Name of a [document-processing](docproc.html) container cluster that does index preprocessing. Use cluster to specify an alternative cluster, other than the default cluster on content nodes.
|
| chain | optional | string | `indexing` chain |
A document processing chain in the container cluster specified by _cluster_ to use for index preprocessing. The chain must inherit the `indexing` chain.
|
Example - the container cluster enables [document-processing](docproc.html), referred to by the content cluster:
```xml
```
To add document processors either before or after the indexer, declare a chain (inherit _indexing_) in a _document-processing_ container cluster and add document processors. Annotate document processors with `before=indexingStart` or `after=indexingEnd`. Configure this cluster and chain as the indexing chain in the content cluster - example:
```xml
indexingStart
indexingEnd
```
**Important:** Note the [document-api](container.html#document-api) configuration. Set up this API on the same nodes as `document-processing` - find details in [indexing](../../../writing/indexing.html).
##### min-redundancy
Contained in [content](#content). The minimum total data copies the cluster will maintain. This can be set instead of (or in addition to) redundancy to ensure that a minimum number of copies are always maintained regardless of other configuration.
On Vespa Cloud, this results in a [redundancy](#redundancy) of at least ceil(min-redundancy / groups) per group; self-managed, a total redundancy of at least min-redundancy. Settings that refer to redundancy, such as [searchable-copies](#searchable-copies), use this derived value. As each group holds at least one full copy, the actual copy count can exceed min-redundancy: on Vespa Cloud, min-redundancy 2 with 3 groups gives 3 copies.
`min-redundancy` can be changed without node restart - replicas will be added or removed automatically.
###### min-redundancy and groups
A group will always have minimum one copy of each document in the cluster. This is also the most commonly used configuration; Increase replica level with more groups to improve query capacity.
- Example 1: If _min-redundancy_ is 2 and there is 1 content group, there will be 2 data copies in the group (2 copies for the cluster). If the number of groups is changed to 2 there will be 1 data copy in each group (still 2 copies for the cluster).
- Example 2: A cluster is configured to [autoscale](../../../operations/autoscaling.html) using `groups="[2,3]"`. Here, configure min-redundancy to 2, as each group will have 1 replica irrespective of number of groups, here 2 or 3 - see [replicas](../../../content/elasticity.html#replicas). Setting the lower bound ensures correct replica level for 2 groups.
For self-managed Vespa: Read more about the actual number of replicas when using [groups](#group) in [topology change](/en/content/elasticity.html#changing-topology).
##### redundancy
Contained in [content](#content).
**Note:** Use [min-redundancy](#min-redundancy) instead of `redundancy`.
Vespa Cloud: The number of data copies _per group_.
Self-managed: The total data copies the cluster will maintain to avoid data loss.
Example: with a redundancy of 2, the system tolerates 1 node failure before data becomes unavailable (until the system has managed to create new replicas on other online nodes).
Redundancy can be changed without node restart - replicas will be added or removed automatically.
##### coverage-policy
Contained in [content](#content).
Specifies the coverage policy for the content cluster. Valid values are `group` or `node`. The default value is `group`.
If the policy is `group` coverage is maintained per group, meaning that when doing maintenance, upgrades etc. one group is allowed to be down at a time. If there is only one group in the cluster, coverage will be the same as policy `node`.
If the policy is `node` coverage is maintained on a node level, meaning that when doing maintenance, upgrades etc. coverage will be maintained on a node level, so in practice 1 node in the whole cluster is allowed to be down at a time.
When having several groups the common reason for changing policy away from the default `group` policy is when the load added to the remaining groups will increase too much when a whole group is allowed to go down. In that case it will be better to use the `node` policy, as taking one node at a time will give just a minor increase in load.
##### node
Contained in [nodes](services.html#nodes) or [group](#group). Configures a content node to the cluster, see [node](services.html#node) in the general services.xml documentation.
Additional node attributes for content nodes:
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| distribution-key | required | integer | |
The unique data distribution id of this node. This **must** remain unchanged for the host's lifetime. Distribution keys of a fresh system should be contiguous and start from zero.
Distribution keys are used to identify nodes and groups for the [distribution algorithm](../../../content/idealstate.html). If a node changes distribution key, the distribution algorithm regards it as a new node, so buckets are redistributed.
|
| capacity | optional | double | 1 |
**Deprecated:** Capacity of this node, relative to other nodes. A node with capacity 2 will get double the data and feed requests of a node with capacity 1. This feature is deprecated and expert mode only. Don't use in production, Vespa assumes homogenous cluster capacity.
|
| baseport | optional | integer | |
baseport The first port in the port range allocated by this node.
|
##### group
Contained in [content](#content) or [group](#group) - groups can be nested. Defines the [hierarchical structure](../../../content/elasticity.html#grouped-distribution) of the cluster. Can not be used in conjunction with the [nodes](services.html#nodes) element. Groups can contain other groups or nodes, but not both. There can only be a single level of leaf groups under the top group.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| distribution-key | required | integer | |
Sets the distribution key of a group. It is not allowed to change this for a given group. Group distribution keys only need to be unique among groups that share the same parent group.
|
| name | required | string | |
The name of the group, used for access from status pages and the like.
|
**Important:** There is no deployment-time verification that the distribution key remains unchanged for any given node or group. Consequently, take great care when modifying the set of nodes in a content cluster. Assigning a new distribution key to an existing node is undefined behavior; Best case, the existing data will be temporarily unavailable until the error has been corrected. Worst case, risk crashes or data loss.
See [Vespa Serving Scaling Guide](../../../performance/sizing-search.html) for when to consider using grouped distribution.
##### distribution (in group)
Contained in [group](#group). Defines the data distribution to subgroups of this group. _distribution_ should not be in the lowest level group containing storage nodes, as here the ideal state algorithm is used directly. In higher level groups, _distribution_ is mandatory.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| partitions | required if there are subgroups in the group | string | | String conforming to the partition specification:
| Partition specification | Description |
| --- | --- |
| \* | Distribute all copies over 1 of N groups |
| 1|\* | Distribute all copies over 2 of N groups |
| 1|1|\* | Distribute all copies over 3 of N groups |
|
The partition specification is used to evenly distribute content copies across groups. Set a number or `*` per group separated by pipes (e.g. `1|*` for two groups). See [sample deployment configurations](../../../operations/self-managed/sizing-examples.html).
##### engine
Contained in [content](#content). Specify the content engine to use, and/or adjust tuning parameters for the engine. Allowed engines are `proton` and `dummy`, the latter being used for debugging purposes. If no engine is given, proton is used. Sub-element: [proton](#proton).
##### proton
Contained in [engine](#engine). If specified, the content cluster will use the Proton content engine. This engine supports storage, indexed search and secondary indices. Optional sub-elements are [searchable-copies](#searchable-copies), [tuning](#tuning-proton), [sync-transactionlog](#sync-transactionlog), [flush-on-shutdown](#flush-on-shutdown), and [resource-limits (in proton)](#resource-limits-proton).
##### searchable-copies
Contained in [proton](#proton). The number of data copies that are indexed (_ready_) and hence searchable.
Vespa Cloud: Searchable copies _per group_, capped at [redundancy](#redundancy). Default 1 with multiple groups, otherwise 2 (or redundancy, if lower).
Self-managed: The total searchable copies, divided evenly across groups - must be divisible by the number of groups. Default 2 or the number of groups if higher, or 1 if redundancy is 1.
In clusters using only [streaming search](../../../performance/streaming-search.html) there is no index, so a searchable copy costs no extra resources - the default is redundancy, making failover instant.
If set to less than redundancy, only some of the stored copies are ready for searching at any time. This means that node failures causes temporary data unavailability while the alternate copies are being indexed for search. The benefit is using less memory, trading off availability during transitions. Refer to [bucket move](../../../content/proton.html#bucket-move).
If updating documents or using [document selection](#documents) for garbage collection, consider setting [fast-access](../../schemas/schemas.html#attribute) on the subset of attribute fields used for this to make sure that these attributes are always kept in memory for fast access. Note that this is only useful if `searchable-copies` is less than `redundancy`. Read more in [proton](../../../content/proton.html).
`searchable-copies` can be changed without node restart. Note that when reducing `searchable-copies` resource usage will not be reduced until content nodes are restarted.
##### tuning
Contained in [proton](#proton), optional. Tune settings for the search nodes in a content cluster - sub-element:
| Element | Required | Quantity |
| --- | --- | --- |
| [searchnode](#searchnode) | No | Zero or one |
##### searchnode
Contained in [tuning](#tuning-proton), optional. Tune settings for search nodes in a content cluster - sub-elements:
| Element | Required | Quantity |
| --- | --- | --- |
| [lidspace](#lidspace) | No | Zero or one |
|
| |
| [requestthreads](#requestthreads) | No | Zero or one |
| [flushstrategy](#flushstrategy) | No | Zero or one |
| [initialize](#initialize) | No | Zero or one |
| [feeding](#feeding) | No | Zero or one |
| [index](#index) | No | Zero or one |
| [summary](#summary) | No | Zero or one |
```xml
```
##### requestthreads
Contained in [searchnode](#searchnode), optional. Tune the number of request threads used on a content node, see [thread-configuration](../../../performance/sizing-search.html#thread-configuration) for details. Sub-elements:
| Element | Required | Default | Description |
| --- | --- | --- | --- |
| search | Optional | **Vespa Cloud:** min(vcpu\*4 + persearch - 1, vcpu\*persearch)
**Self-hosted:** 64. |
Total size of the match engine thread pool. Together with `persearch`, this determines the maximum number of queries that can execute concurrently: `search / persearch`. See the [Vespa serving scaling guide](../../../performance/sizing-search.html#thread-configuration) for sizing guidance.
|
| persearch | Optional | 1 |
Maximum number of threads used per search. A higher value reduces the time queries spend in query evaluation, except time spent in ANN which is single-threaded. This number of threads is held for each query for the duration of the query, also when much of the time is spent on single-threaded operations. See the [Vespa serving scaling guide](../../../performance/sizing-search.html) for an introduction of using multiple threads per search per node to reduce query latency. Number of threads per search can be adjusted down per _rank-profile_ using [num-threads-per-search](../../schemas/schemas.html#num-threads-per-search).
|
| summary | Optional | **Vespa Cloud:** vcpu **Self-hosted:** 16 |
Number of summary threads.
|
| resultprocessing | Optional | No limit |
Maximum number of [search](#requestthreads-search) threads that can do memory-hungry result processing at the same time. This applies to [sorting](../../querying/sorting-language.html) (including sorting on [sort-features](../../schemas/schemas.html#sort-features)) and [grouping](../../querying/grouping-language.html) that are likely to require large amounts of memory. Threads exceeding the limit wait until another thread has completed its result processing. Use this to limit peak memory usage on content nodes when many concurrent queries sort or group over large result sets. To have any effect, the value should be at least 1 and less than the `search` value.
|
```xml
64
1
16
8
```
##### flushstrategy
Contained in [searchnode](#searchnode), optional. Tune the _native_-strategy for flushing components to disk - a smaller number means more frequent flush:
- _Memory gain_ is how much memory can be freed by flushing a component
- _Disk gain_ is how much disk space can be freed by flushing a component (typically by using compaction)
Refer to [Proton maintenance jobs](../../../content/proton.html#proton-maintenance-jobs). Optional sub-elements:
- `native`:
- `total`
- `maxmemorygain`: The total maximum memory gain (in bytes) for _all_ components before running flush, default 4294967296 (4 GB)
- `diskbloatfactor`: Trigger flush if the total disk gain (in bytes) for _all_ components is larger than the factor times current total disk usage, default 0.25
- `component`
- `maxmemorygain`: The maximum memory gain (in bytes) by _a single_ component before running flush, default 1073741824 (1 GB)
- `diskbloatfactor`: Trigger flush if the disk gain (in bytes) by _a single_ component is larger than the given factor times the current disk usage by that component, default 0.25
- `maxage`: The maximum age (in seconds) of unflushed content for a single component before running flush, default 111600 (31h)
- `transactionlog`
- `maxsize`: The total maximum size (in bytes) of [transaction logs](../../../content/proton.html#transaction-log) for all document types before running flush, default 21474836480 (20 GB)
- `conservative`
- `memory-limit-factor`: When [resource-limits (in proton)](#resource-limits-proton) for memory is reached, flush more often by downscaling `total.maxmemorygain` and `component.maxmemorygain`, default 0.5
- `disk-limit-factor`: When [resource-limits (in proton)](#resource-limits-proton) for disk is reached, flush more often by downscaling `transactionlog.maxsize`, default 0.5
```xml
4294967296
0.2
1073741824
0.2
111600
21474836480
0.5
0.5
```
##### initialize
Contained in [searchnode](#searchnode), optional. Tune settings related to how the search node (proton) is initialized. Optional sub-elements:
- `threads`: The number of initializer threads used for loading structures from disk at proton startup. The threads are shared between document databases when the value is larger than 0. Default value is the number of document databases + 1.
- When set to larger than 1, document databases are initialized in parallel
- When set to 1, document databases are initialized in sequence
- When set to 0, 1 separate thread is used per document database, and they are initialized in parallel.
```xml
2
```
##### lidspace
Contained in [searchnode](#searchnode), optional. Tune settings related to how lidspace is managed. Optional sub-elements:
- `max-bloat-factor`: Maximum bloat allowed before lidspace compaction is started. Compaction is moving a document from a high lid to a lower lid. Cost is similar to feeding a document and removing it. Also see description in [lidspace compaction maintenance job](../../../content/proton.html#lid-space-compaction). Default value is 0.01 or 1% of total lidspace. Will be increased to target of 0.50 or 50%.
```xml
0.5
```
##### feeding
Contained in [searchnode](#searchnode), optional. Tune [proton](../../../content/proton.html) settings for feed operations. Optional sub-elements:
- `concurrency`: A number between 0.0 and 1.0 that specifies the concurrency when handling feed operations, default 0.5. When set to 1.0, all cores on the cpu can be used for feeding. Changing this value requires a restart of the nodes to take effect.
- `niceness`: A number between 0.0 and 1.0 that specifies the niceness of the feeding threads, default 0.0 =\> not any nicer than anyone else. Increasing this number will reduce the priority of feeding compared to search. The real-world effect is hard to predict as the magic exists in the OS level scheduler. Changing this value requires a restart of the nodes to take effect.
```xml
0.8
0.5
```
Note: Queries can always use _all_ available cores; concurrency caps how many can be used for writes and maintenance, which includes redistribution.
##### index
Contained in [searchnode](#searchnode), optional. Tune various aspect with the handling of disk and memory indexes. Optional sub-elements:
- `io`
- `search`: Controls io read options used during search, values={mmap,populate}, default `mmap`. Using `populate` will eagerly touch all pages when index is loaded (after re-start or after index fusion is complete).
- `warmup`
- `time`: Specifies in seconds how long the index shall be warmed up before being switched in for serving. During warmup, it will receive queries and posting lists will be iterated, but results ignored as they are duplicates of the live index. This will pull in the most important ones in the cache. However, as warming up an index will occupy more memory, do not turn it on unless you suspect you need it. And always benchmark to see if it is worth it.
- `unpack`: Controls whether all posting features are pulled in to the cache, or only the most important. values={true, false}, default false.
```xml
mmap
true
```
##### removed-db
Contained in [searchnode](#searchnode), optional. Tune various aspect of the db of removed documents. Optional sub-elements:
- `prune`
- `age`: Specifies how long (in seconds) we must remember removed documents before we can prune them away. Default is 2 weeks. This sets the upper limit on how long a node can be down and still be accepted back in the system, without having the index wiped. There is no point in having this any higher than the age of the documents. If corpus is re-fed every day, there is no point in having this longer than 24 hours.
- `interval`: Specifies how often (in seconds) to prune old documents. Default is 3.36 hours (prune age / 100). No need to change default. Exposed here for reference and for testing.
```xml
86400
```
##### summary
Contained in [searchnode](#searchnode), optional. Tune various aspect with the handling of document summary. Optional sub-elements:
- `io`
- `read`: Controls io read options used during reading of stored documents. Values are `directio` `mmap` `populate`. Default is `mmap`. `populate` will do an eager mmap and touch all pages.
- `store`
- `cache`: Used to tune the cache used by the document store. Enabled by default, using up to 5% of available memory.
- `maxsize`: The maximum size of the cache in bytes. If set, it takes precedence over [maxsize-percent](#summary-store-cache-maxsize-percent). Default is unset.
- `maxsize-percent`: The maximum size of the cache in percent of available memory. Default is 5%.
- `compression`
- `type`: The compression type of the documents while in the cache. Possible values are , `none` `lz4` `zstd`. Default is `lz4`
- `level`: The compression level of the documents while in cache. Default is 6
- `logstore`: Used to tune the actual document store implementation (log-based).
- `maxfilesize`: The maximum size (in bytes) per summary file on disk. Default value is 1GB. [document-store-compaction](../../../content/proton.html#document-store-compaction)
- `chunk`
- `maxsize`: Maximum size (in bytes) of a chunk. Default value is 64KB.
- `compression`
- `type`: Compression type for the documents, `none` `lz4` `zstd`. Default is `zstd`.
- `level`: Compression level for the documents. Default is 3.
```xml
directio
5
none
16384
zstd
3
```
##### flush-on-shutdown
Contained in [proton](#proton). Default value is true. If set to true, search nodes will flush a set of components (e.g. memory index, attributes) to disk before shutting down such that the time it takes to flush these components plus the time it takes to replay the [transaction log](../../../content/proton.html#transaction-log) after restart is as low as possible. The time it takes to replay the transaction log depends on the amount of data to replay, so by flushing, some components before restart the transaction log will be pruned, and we reduce the replay time significantly. Refer to [Proton maintenance jobs](../../../content/proton.html#proton-maintenance-jobs).
##### sync-transactionlog
Contained in [proton](#proton). Default value is true. If true, the transactionlog is synced to disk after every write. This enables the transactionlog to survive power failures and kernel panic. The sync cost is amortized over multiple feed operations. The faster you feed the more operations it is amortized over. So with a local disk this is not known to be a performance issue. However, if using NAS (Network Attached Storage) like EBS on AWS one can see significant feed performance impact. For one particular case, turning off sync-transactionlog for EBS gave a 60x improvement.
With sync-transactionlog turned off, the risk of losing data depends on the kernel's [sysctl settings.](https://www.kernel.org/doc/html/latest/admin-guide/sysctl/vm.html#dirty-background-bytes) For example, this is a common default:
```
#### sysctl -a
...
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
...
```
With this configuration, the worse case scenario is to lose 35 seconds worth of transactionlog, but no more than 1/20 of the free memory. Because kernel flusher threads wake up every 5s (dirty\_writeback\_centisecs) and write data older than 30s (dirty\_expire\_centisecs) from memory to disk. But if un-synced data exceeds 1/20 of the free memory, the Vespa process will sync it (dirty\_ratio).
The above also assumes that all copies of the data are lost at the same time **and** that kernels on all these nodes flush at the same time: realistic scenario only with one copy.
Adjust these [sysctl settings](https://www.kernel.org/doc/html/latest/admin-guide/sysctl/vm.html#dirty-background-bytes) to manage the trade-off between data loss and performance. You'll see more in those kernel docs: for example, thresholds can be expressed in bytes.
##### resource-limits (in proton)
Contained in [proton](#proton). Specifies resource limits used by proton to reject both external and internal write operations (on this content node) when a limit is reached.
**Warning:** These proton limits should almost never be changed directly. Instead, change [resource-limits](#resource-limits) that controls when external write operations are blocked in the entire content cluster. Be aware of the risks of tuning resource limits as seen in the link.
The local proton limits are derived from the cluster limits if not specified, using this formula:
$${L\_{proton}} = {L\_{cluster}} + \frac{1-L\_{cluster}}{2}$$
| Element | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| disk | optional | float
[0, 1] | 0.9 |
Fraction of total space on the disk partition used before put and update operations are rejected
|
| memory | optional | float
[0, 1] | 0.9 |
Fraction of physical memory that can be resident memory in anonymous mapping by proton before put and update operations are rejected
|
Example:
```xml
0.83
0.82
```
##### search
Contained in [content](#content), optional. Declares search configuration for this content cluster. Optional sub-elements are [query-timeout](#query-timeout), [visibility-delay](#visibility-delay) and [coverage](#coverage).
##### query-timeout
Contained in [search](#search). Specifies the query timeout in seconds for queries against the search interface on the content nodes. The default is 0.5 (500ms), the max is 600.0. For query timeout also see the request parameter [timeout](../../api/query.html#timeout).
**Note:** One can not override this value using the [timeout](../../api/query.html#timeout) request parameter.
##### visibility-delay
Contained in [search](#search). Default 0, max 1, seconds.
This setting controls the TTL caching for [parent-child](../../../schemas/parent-child.html) imported fields. See [feature tuning](../../../performance/feature-tuning.html#parent-child-and-search-performance).
##### coverage
Contained in [search](#search). Declares search coverage configuration for this content cluster. Optional sub-elements are [minimum](#minimum), [min-wait-after-coverage-factor](#min-wait-after-coverage-factor) and [max-wait-after-coverage-factor](#max-wait-after-coverage-factor). Search coverage configuration controls how many nodes the query dispatcher process should wait for, trading search coverage versus search performance.
##### minimum
Contained in [coverage](#coverage). Declares the minimum search coverage required before returning the results of a query. This number is in the range `[0, 1]`, with 0 being no coverage and 1 being full coverage.
The default is 1; unless configured otherwise a query will not return until all search nodes have responded within the specified timeout.
##### min-wait-after-coverage-factor
Contained in [coverage](#coverage). Declares the minimum time for a query to wait for full coverage once the declared [minimum](#minimum) has been reached. This number is a factor that is multiplied with the time remaining at the time of reaching minimum coverage.
The default is 0; unless configured otherwise a query will return as soon as the minimum coverage has been reached, and the remaining search nodes appear to be lagging.
##### max-wait-after-coverage-factor
Contained in [coverage](#coverage). Declares the maximum time for a query to wait for full coverage once the declared [minimum](#minimum) has been reached. This number is a factor that is multiplied with the time remaining at the time of reaching minimum coverage.
The default is 1; unless configured otherwise a query is allowed to wait its full timeout for full coverage even after reaching the minimum.
##### tuning
Contained in [content](#content), optional. Optional tuning parameters are: [bucket-splitting](#bucket-splitting), [min-node-ratio-per-group](#min-node-ratio-per-group), [cluster-controller](#cluster-controller), [dispatch](#dispatch-tuning), [distribution](#distribution_type), [max-document-size](#max-document-size), [merges](#merges), [persistence-threads](#persistence-threads) and [visitors](#visitors).
##### bucket-splitting
Contained in [tuning](#tuning). The [bucket](../../../content/buckets.html) is the fundamental unit of distribution and management in a content cluster. Buckets are auto-split, no need to configure for most applications.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| max-documents | optional | integer | 1024 |
Maximum number of documents per content bucket. Buckets are split in two if they have more documents than this. Keep this value below 16K.
|
| max-size | optional | integer | 32MiB |
Maximum size (in bytes) of a bucket. This is the sum of the serialized size of all documents kept in the bucket. Buckets are split in two if they have a larger size than this. Keep this value below 100 MiB.
|
| minimum-bits | optional | integer | |
Override the ideal distribution bit count configured for this cluster. Prefer to use the [distribution type](#distribution_type) setting instead if the default distribution bit count does not fit the cluster. This variable is intended for testing and to work around possible distribution bit issues. Most users should not need this option.
|
##### min-node-ratio-per-group
**Important:** This is configuration for the cluster controller. Most users are normally looking for [min-active-docs-coverage](#min-active-docs-coverage) which controls how many nodes can be down before query load is routed to other groups.
Contained in [tuning](#tuning). States a lower bound requirement on the ratio of nodes within _individual_ [groups](#group) that must be online and able to accept traffic before the entire group is automatically taken out of service. Groups are automatically brought back into service when the availability of its nodes has been restored to a level equal to or above this limit.
Elastic content clusters are often configured to use multiple groups for the sake of horizontal traffic scaling and/or data availability. The content distribution system will try to ensure a configured number of replicas is always present within a group in order to maintain data redundancy. If the number of available nodes in a group drops too far, it is possible for the remaining nodes in the group to not have sufficient capacity to take over storage and serving for the replicas they now must assume responsibility for. Such situations are likely to result in increased latencies and/or feed rejections caused by resource exhaustion. Setting this tuning parameter allows the system to instead automatically take down the remaining nodes in the group, allowing feed and query traffic to fail completely over to the remaining groups.
Valid parameter is a decimal value in the range [0, 1]. Default is 0, which means that the automatic group out-of-service functionality will _not_ automatically take effect.
Example: assume a cluster has been configured with _n_ groups of 4 nodes each and the following tuning config:
```xml
0.75
```
This tuning allows for 1 node in a group to be down. If 2 or more nodes go down, all nodes in the group will be marked as down, letting the _n-1_ remaining groups handle all the traffic.
This configuration can be changed live as the system is running and altered limits will take effect immediately.
##### distribution (in tuning)
Contained in [tuning](#tuning). Tune the distribution algorithm used in the cluster.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| type | optional | loose | strict | legacy | loose |
When the number of a nodes configured in a system changes over certain limits, the system will automatically trigger major redistributions of documents. This is to ensure that the number of buckets is appropriate for the number of nodes in the cluster. This enum value specifies how aggressive the system should be in triggering such distribution changes.
The default of `loose` strikes a balance between rarely altering the distribution of the cluster and keeping the skew in document distribution low. It is recommended that you use the default mode unless you have empirically observed that it causes too much skew in load or document distribution.
Note that specifying `minimum-bits` under [bucket-splitting](#bucket-splitting) overrides this setting and effectively "locks" the distribution in place.
|
##### max-document-size
Contained in [tuning](#tuning). Specifies max document size in the content cluster, measured as the uncompressed size of a document operation arriving over the wire by the distributor service. The limit will be used for all document types. A document larger than this limit will be rejected by the distributor. Note that some document operations that don't contain the entire document, like [document updates](../../../writing/document-api-guide.html#document-updates) might increase the size of a document above this limit. See also [max-document-size](container.html#document-api) for document api, which is probably what you want to configure, make sure to keep the value documented here larger.
Valid values are numbers including a unit (e.g. _10MiB_) and the value must be between 1Mib and 2048 Mib (inclusive). Values will be rounded to nearest MiB, so using MiB as a unit is preferrable. It is strongly recommended to make sure this is not set too high, 10 MiB is a reasonable setting for most use cases, setting it above 100 MiB is not recommended, as allowing large documents might impact operations, e.g. when restarting nodes, moving documents between nodes etc. Default value is 128 MiB.
Example:
```xml
10MiB
```
##### merges
Contained in [tuning](#tuning). Defines throttling parameters for bucket merge operations.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| max-per-node | optional | number | | Maximum number of parallel active bucket merge operations. |
| max-queue-size | optional | number | | Maximum size of the merge bucket queue, before reporting BUSY back to the distributors. |
##### persistence-threads
Contained in [tuning](#tuning). Defines the number of persistence threads per partition on each content node. A content node executes bucket operations against the persistence engine synchronously in each of these threads. 8 threads are used by default. Override with the **count** attribute.
##### visitors
Contained in [tuning](#tuning). Tuning parameters for visitor operations. Might contain [max-concurrent](#max-concurrent).
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| thread-count | optional | number | | The maximum number of threads in which to execute visitor operations. A higher number of threads may increase performance, but may use more memory. |
| max-queue-size | optional | number | | Maximum size of the pending visitor queue, before reporting BUSY back to the distributors. |
##### max-concurrent
Contained in [visitors](#visitors). Defines how many visitors can be active concurrently on each storage node. The number allowed depends on priority - lower priority visitors should not block higher priority visitors completely. To implement this, specify a fixed and a variable number. The maximum active is calculated by adjusting the variable component using the priority, and adding the fixed component.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| fixed | optional | number | [16](https://github.com/vespa-engine/vespa/blob/master/storage/src/vespa/storage/visiting/stor-visitor.def) | The fixed component of the maximum active count |
| variable | optional | number | [64](https://github.com/vespa-engine/vespa/blob/master/storage/src/vespa/storage/visiting/stor-visitor.def) | The variable component of the maximum active count |
##### resource-limits
Contained in [tuning](#tuning). Specifies resource limits used to decide whether external write operations should be blocked in the entire content cluster, based on the reported resource usage by content nodes. See [feed block](../../../writing/feed-block.html) for more details.
**Warning:** The content nodes require resource headroom to handle extra documents as part of re-distribution during node failure, and spikes when running [maintenance jobs](../../../content/proton.html#proton-maintenance-jobs). Tuning these limits should be done with extreme care, and setting them too high might lead to permanent data loss. They are best left untouched, using the defaults, and cannot be set in Vespa Cloud.
| Element | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| disk | optional | float
[0, 1] | 0.8 |
Fraction of total space on the disk partition used on a content node before feed is blocked
|
| memory | optional | float
[0, 1] | 0.8/0.75 |
Fraction of physical memory that can be resident memory in anonymous mapping on a content node before feed is blocked. Total physical memory is sampled as the minimum of `sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGESIZE)` and the cgroup (v1 or v2) memory limit. Nodes with 8 Gib or less memory in Vespa Cloud has a limit of 0.75.
|
Example - in the content tag:
```xml
0.78
0.77
```
##### dispatch
Contained in [tuning](#tuning). Tune the query dispatch behavior - child elements:
| Element | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| max-hits-per-partition | optional | Integer | No capping: Return all |
Maximum number of hits to return from a content node. By default, a query returns the requested number of hits + offset from every content node to the container. The container orders the hits globally according to the query, then discards all hits beyond the number requested.
In a system with a large fan-out, this consumes network bandwidth and the container nodes easily network saturated. Containers will also sort and discard more hits than optimal.
When there are sufficiently many search nodes, assuming an even distribution of the hits, it suffices to only return a fraction of the request number of hits from each node. Note that changing this number will have global ordering impact. See _top-k-probability_ below for improving performance with fewer hits.
|
| dispatch-policy | optional | adaptive / best-of-random-2 / round-robin | adaptive |
With [grouped distribution](../../../performance/sizing-search.html#data-distribution): Configure policy for choosing which group shall receive the next query request. Coverage requirements is considered when choosing a group. Note that multiphase requests that requires or benefits from hitting the same group in all phases are always hashed.
| adaptive | Measures latency, preferring lower latency groups, selecting group `i` has a probability proportional to 1 / (latency for group `i`). |
| best-of-random-2 | Selects 2 random groups and selects the one with the lowest latency. |
| round-robin | Selects groups in a round-robin manner, giving fair distribution of queries to each group. |
|
| prioritize-availability | optional | Boolean | true |
With [grouped distribution](../../../performance/sizing-search.html#data-distribution): If true, or by default, all groups that are within min-active-docs-coverage of the **median** of the document count of other groups will be used to service queries. If set to false, only groups within min-active-docs-coverage of the **max** document count will be used, with the consequence that full coverage is prioritized over availability when multiple groups are lacking content, since the remaining groups may not be able to service the full query load.
|
| min-active-docs-coverage | optional | A float percentage | 97 |
With [grouped distribution](../../../performance/sizing-search.html#data-distribution): The percentage of active documents a group must have, relative to the median across all groups in the content cluster, to be considered active for serving queries. Because of measurement timing differences, it is not advisable to tune this above 99 percent.
|
| top-k-probability | optional | Double | 0.9999 |
Probability that the top K hits will be the globally best. Based on this probability, the dispatcher will fetch enough hits from each node to achieve this. The only way to guarantee a probability of 1.0 is to fetch K hits from each partition. However, by reducing the probability from 1.0 to 0.99999, one can significantly reduce number of hits fetched and save both bandwidth and latency. The number of hits to fetch from each partition is computed as:
$${q}={\frac{k}{n}}+{qT}({p},{30})×{\sqrt{ {k}×{\frac{1}{n}}×({1}-{\frac{1}{n}}) }}$$
where qT is a Student's t-distribution. With n=10 partitions, k=200 hits and p=0.99999, only 45 hits per partition is needed, as opposed to 200 when p=1.0.
Use this option to reduce network and container cpu/memory in clusters with many nodes per group - see [Vespa Serving Scaling Guide](../../../performance/sizing-search.html).
|
##### cluster-controller
Contained in [tuning](#tuning). Tuning parameters for the cluster controller managing this cluster - child elements:
| Element | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| init-progress-time | optional | | |
If the initialization progress count have not been altered for this amount of seconds, the node is assumed to have deadlocked and is set down. Note that initialization may actually be prioritized lower now, so setting a low value here might cause false positives. Though if it is set down for wrong reason, when it will finish initialization and then be set up again.
|
| transition-time | optional | | [storage\_transition\_time](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/fleetcontroller.def)[distributor\_transition\_time](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/fleetcontroller.def) |
The transition time states how long (in seconds) a node will be in maintenance mode during what looks like a controlled restart. Keeping a node in maintenance mode during a restart allows a restart without the cluster trying to create new copies of all the data immediately. If the node has not started or got back up within the transition time, the node is set down, in which case, new full bucket copies will be created. Note separate defaults for distributor and storage (i.e. search) nodes.
|
| max-premature-crashes | optional | | [max\_premature\_crashes](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/fleetcontroller.def) |
The maximum number of crashes allowed before a content node is permanently set down by the cluster controller. If the node has a stable up or down state for more than the _stable-state-period_, the crash count is reset. However, resetting the count will not re-enable the node again if it has been disabled - restart the cluster controller to reset.
|
| stable-state-period | optional | | [stable\_state\_time\_period](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/fleetcontroller.def) |
If a content node's state doesn't change for this many seconds, it's state is considered _stable_, clearing the premature crash count.
|
| min-distributor-up-ratio | optional | | [min\_distributor\_up\_ratio](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/fleetcontroller.def) |
The minimum ratio of distributors that are required to be _up_ for the cluster state to be _up_.
|
| min-storage-up-ratio | optional | | [min\_storage\_up\_ratio](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/fleetcontroller.def) |
The minimum ratio of content nodes that are required to be _up_ for the cluster state to be _up_.
|
| groups-allowed-down-ratio | optional | | [groups-allowed-down-ratio](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/fleetcontroller.def) |
A ratio for the number of content groups that are allowed to be down simultaneously. A value of 0.5 means that 50% of the groups are allowed to be down. The default is to allow only one group to be down at a time.
|
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [services.xml - 'content'](#page-title)
- [content](#content)
- [documents](#documents)
- [document](#document)
- [document-processing](#document-processing)
- [min-redundancy](#min-redundancy)
- [min-redundancy and groups](#min-redundancy-and-groups)
- [redundancy](#redundancy)
- [coverage-policy](#coverage-policy)
- [node](#node)
- [group](#group)
- [distribution (in group)](#distribution)
- [engine](#engine)
- [proton](#proton)
- [searchable-copies](#searchable-copies)
- [tuning](#tuning-proton)
- [searchnode](#searchnode)
- [requestthreads](#requestthreads)
- [flushstrategy](#flushstrategy)
- [initialize](#initialize)
- [lidspace](#lidspace)
- [feeding](#feeding)
- [index](#index)
- [removed-db](#removed-db)
- [summary](#summary)
- [flush-on-shutdown](#flush-on-shutdown)
- [sync-transactionlog](#sync-transactionlog)
- [resource-limits (in proton)](#resource-limits-proton)
- [search](#search)
- [query-timeout](#query-timeout)
- [visibility-delay](#visibility-delay)
- [coverage](#coverage)
- [minimum](#minimum)
- [min-wait-after-coverage-factor](#min-wait-after-coverage-factor)
- [max-wait-after-coverage-factor](#max-wait-after-coverage-factor)
- [tuning](#tuning)
- [bucket-splitting](#bucket-splitting)
- [min-node-ratio-per-group](#min-node-ratio-per-group)
- [distribution (in tuning)](#distribution_type)
- [max-document-size](#max-document-size)
- [merges](#merges)
- [persistence-threads](#persistence-threads)
- [visitors](#visitors)
- [max-concurrent](#max-concurrent)
- [resource-limits](#resource-limits)
- [dispatch](#dispatch-tuning)
- [cluster-controller](#cluster-controller)
---
### deployment.xml reference
[](/en/reference/applications/deployment.html.md "View as Markdown")
#### deployment.xml reference
[](/en/reference/applications/deployment.html.md "View as Markdown")
_deployment.xml_ controls how an application is deployed.
_deployment.xml_ is placed in the root of the [application package](../../basics/applications.html) and specifies which environments and regions the application is deployed to during [automated application deployment](../../operations/automated-deployments.html), as which application instances.
Deployment progresses through the `test` and `staging` environments to the `prod` environments listed in _deployment.xml_.
Simple example:
```xml
```
More complex example:
```xml
aws-us-west-2a
beta
```
Some of the elements can be declared _either_ under the `` root, **or**, if one or more `` tags are listed, under these. These have a bold **or** when listing where they may be present.
##### deployment
The root element.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| version | Yes | 1.0 |
| major-version | No | The major version number this application is valid for. |
| cloud-account | No | Account to deploy to with [Vespa Cloud Enclave](../../operations/enclave/enclave). |
| empty-host-ttl | No | How long to keep hosts after they become empty, with [Vespa Cloud Enclave](../../operations/enclave/enclave), e.g., `30m`, `6h` or `2d`. See [empty host TTL](#empty-host-ttl). |
**Empty host TTL.** With a `cloud-account`, hosts are provisioned for each deployment, and removed shortly after they become empty. Deployments to the `test` and `staging` environments are removed after each run, so every system and staging test run waits for new hosts. Set `empty-host-ttl` to keep empty hosts for that long instead; a later deployment of the same instance in the same zone, including its tester, then reuses them, as long as the node resources are unchanged. The value is an integer followed by `m`, `h` or `d`, and `0m` turns it off. It is inherited from the closest enclosing element, and has no effect without a cloud account. Manual deployments to `dev` use the value on ``, or else the one on ``.
To keep hosts only for system and staging tests, set it on `` or ``, and turn it off for production, and for `dev` when set on ``. Don't add `` or `` elements only to set it there, as tests are then required in the test package, and runs without them fail:
```xml
aws-us-east-1c
```
In production, the TTL keeps hosts emptied by scaling down, which makes a later scale-up with the same resources faster, at the cost of the idle hosts.
##### instance
In `` or `` (which must be a direct descendant of the root). An instance of the application; several of these may be simultaneously deployed in the same zone. If no `` is specified, all children of the root are implicitly children of an `` with `id="default"`, as in the simple example at the top.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| id | Yes | The unique name of the instance. |
| tags | No | Space-separated tags which can be referenced to make [deployment variants](../../operations/deployment-variants.html). |
| cloud-account | No | Account to deploy to with [Vespa Cloud Enclave](../../operations/enclave/enclave). Overrides parent's use of cloud-account. |
| empty-host-ttl | No | How long to keep hosts after they become empty. Overrides parent's use of empty-host-ttl. See [empty host TTL](#empty-host-ttl). |
##### block-change
In ``, **or** ``. This blocks changes from being deployed to production in the matching time interval. Changes are nevertheless tested while blocked.
By default, both application revision changes and Vespa platform changes (upgrades) are blocked. It is possible to block just one kind of change using the `revision` and `version` attributes.
Any combination of the attributes below can be specified. Changes on a given date will be blocked if all conditions are met. Invalid `` tags (i.e. that contains conditions that never match an actual date) are rejected by the system.
This tag must be placed after any `` and `` tags, and before ``. It can be declared multiple times.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| revision | No, default `true` | Set to `false` to allow application deployments |
| version | No, default `true` | Set to `false` to allow Vespa platform upgrades |
| maintenance | No, default `false` | Set to `true` to disallow Vespa maintenance operations. This is best effort, maintenance can still happen (e.g. for security reasons). The block window for maintenance should be open at least 10% of the time calculated over a week, that is, at least 17 hours per week. |
| days | No, default `mon-sun` | List of days this block is effective - a comma-separated list of single days or day intervals where the start and end day are separated by a dash and are inclusive. Each day is identified by its english name or three-letter abbreviation. |
| hours | No, default `0-23` | List of hours this block is effective - a comma-separated list of single hours or hour intervals where the start and end hour are separated by a dash and are inclusive. Each hour is identified by a number in the range 0 to 23. |
| time-zone | No, default UTC | The name of the time zone used to interpret the hours attribute, with daylight saving time (DST) transitions taken into account. Time zones are full names or short forms, when the latter is unambiguous. See [ZoneId.of](https://docs.oracle.com/javase/8/docs/api/java/time/ZoneId.html#of-java.lang.String-) for the full spec of acceptable values. |
| from-date | No | The inclusive starting date of this block (ISO-8601, `YYYY-MM-DD`), optionally followed by a time of day, `Thh:mm`. |
| to-date | No | The inclusive ending date of this block (ISO-8601, `YYYY-MM-DD`), optionally followed by a time of day, `Thh:mm`. |
The below example blocks all changes on weekends, and blocks revisions outside working hours, in the PST time zone:
```xml
```
To block _only_ maintenance operations, `revision` and `version` must be explicitly set to `false`, since they default to `true`. The below example blocks maintenance during working hours, while still allowing application deployments and platform upgrades at any time:
```xml
```
The below example blocks:
- all changes on Sundays starting on 2022-03-01
- all changes in the hours 16-23 between 2022-02-10 and 2022-02-15
- all changes until 2022-01-05T20:20
```xml
```
Add `` children to limit a block to the given production regions. Other regions deploy as normal. Regions must be declared in ``. The below example blocks revisions in `aws-us-east-1c` during working hours:
```xml
aws-us-east-1c
```
##### backup
In ``, **or** ``. Configures scheduled backups of production content clusters. When present, backups will be created at the specified frequency. Must be placed after any `` and `` tags, and before ``.
Note that the first backup is not created immediately when this element is added. A cluster becomes eligible for its first backup only once a content node has been running for at least one full `frequency` interval; subsequent backups then follow at the configured frequency. See [Automated Backups](../../operations/data-management.html#backup) for details.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| frequency | Yes | A positive integer with a suffix `h` (hours) or `d` (days), e.g. `24h` or `7d`. Minimum 24h. |
| granularity | No, default `cluster` |
- `cluster`: all content nodes in the cluster
- `group`: all content nodes in a single group
|
Backup activity does not affect service availability, but has costs in terms of performance. You can use `granularity` to control the tradeoff between backup and restoration speed.
- A `cluster` backup will take longer, as each content node must be temporarily suspended to ensure data integrity. Restoration will however require effectively zero content redistribution.
- A `group` backup will be faster, as an entire group will be suspended and backed up simultaneously. Restoration may however require a significant amount of content redistribution, depending on the cluster topology.
In most situations we recommend `cluster` backups.
Vespa Cloud will always keep the three latest backups. Older backups will expire and be removed after the backup interval, e.g. after 21 days for a 7-day frequency, given that there are more recent fully completed backups.
If a cluster or application has been removed, its backups are kept for 30 days after the last completed backup. If backups are turned off for a cluster, its backups are deleted as soon as they expire.
[Block windows](#block-change) also prevent new backups from starting in the given period. If the available time is too short for a full backup to complete, the process will, however, extend beyond the block window.
[Read more](../../operations/data-management.html#backup).
##### resource-tags
In ``, **or** ``. Specifies custom tags to apply to cloud resources (virtual machines and attached disks) provisioned in the tenant cloud account. Only available for [Vespa Cloud Enclave](../../operations/enclave/enclave) deployments, where a `cloud-account` is set. Commonly used for cost tracking and resource management.
Tags declared at the `` level apply to all instances. Tags at the `` level are merged with deployment-level tags; on key conflict, the instance-level value wins.
```xml
```
The `` element contains one or more `` children. Each `` has two mandatory attributes:
| Attribute | Mandatory | Values |
| --- | --- | --- |
| key | Yes | The tag key. Must be non-empty. Allowed characters and maximum length depend on the target cloud; see [per-cloud rules](#resource-tags-per-cloud-rules) below. Certain keys are [reserved](#resource-tags-reserved-keys) by the platform. |
| value | Yes | The tag value. Must be non-empty. May contain [template variables](#resource-tags-template-variables). Allowed characters and maximum length depend on the target cloud; see [per-cloud rules](#resource-tags-per-cloud-rules) below. |
A maximum of 20 tags may be defined per instance (after merging deployment-level and instance-level tags). This limit is the same for all clouds; the remaining tag capacity at each provider is reserved for platform-owned tags.
**Per-cloud rules.** Allowed characters and length limits vary by cloud provider. A single deployment can span multiple clouds, so tags are validated against the rules of each target cloud at deploy time. If a tag is valid for AWS but not for GCP, the deployment will succeed in AWS regions but fail in GCP regions.
| Constraint | AWS | Azure | GCP |
| --- | --- | --- | --- |
| Key characters | `[a-zA-Z0-9 +-=._:/@]` | Unicode, except `< > % & \ ? /` | `[a-z][a-z0-9_-]*` (must start with lowercase letter) |
| Value characters | `[a-zA-Z0-9 +-=._:/@]` | No restrictions | `[a-z0-9_-]*` |
| Key max length | 128 | 512 | 63 |
| Value max length | 256 | 256 | 63 |
**Reserved keys.** The following tag keys are reserved by the platform and cannot be used. All comparisons are case-insensitive.
| Type | Reserved keys |
| --- | --- |
| Exact names | `app`, `application`, `applicationid`, `athenz`, `athenz-domain`, `athenzservice`, `auth-method`, `cluster`, `clusterid`, `fqdn`, `generation`, `name`, `owner`, `preprovisioned`, `system`, `tenant`, `tenantName`, `zone` |
| Key prefixes | `vai_`, `corp_`, `corp:`, `bastion_`, `bastion:` |
**Template variables.** Tag values may reference the following template variables. Resolved values are always lowercased regardless of cloud. Template-variable placeholders are excluded when checking per-cloud character rules, so only the literal parts of the value are validated. Referencing an unknown variable causes the deployment to fail. Variables can be combined, e.g. `value="${environment}-${clustertype}"`.
| Variable | Description |
| --- | --- |
| `${tenant}` | The tenant name, e.g. `mytenant`. |
| `${application}` | The application name, e.g. `myapp`. |
| `${instance}` | The instance name, e.g. `default`, `beta`. |
| `${environment}` | The deployment environment, e.g. `prod`, `dev`. |
| `${region}` | The deployment region, e.g. `aws-us-east-1c`. |
| `${clustername}` | The cluster ID from [services.xml](services/services.html), e.g. `default`, `music`. |
| `${clustertype}` | The Vespa cluster type: `container`, `content`, or `admin`. |
**Reconciliation.** Tags are applied to virtual machines and attached disks. When tags are changed, added, or removed in _deployment.xml_, the existing resources are updated by a background reconciliation process. Tags that were previously applied by Vespa Cloud but are no longer listed are removed from the resources. Tags added manually by the tenant in the cloud console are preserved.
##### upgrade
In ``, or ``. Determines the strategy for upgrading the application, or one of its instances. By default, application revision changes deploy independently of platform upgrades, and an application revision can catch up to and pass an ongoing platform upgrade. See the `rollout` attribute below to change this behavior.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| rollout | No, default `simultaneous` |
- `separate`: When a revision catches up to a platform upgrade, it stays behind, unless the upgrade alone fails.
- `leading`: When a revision catches up to a platform upgrade, they fuse and roll out together.
- `simultaneous` is the default, and favors revision roll-out. Revision changes deploy independently of platform upgrades. When a revision catches up to a platform upgrade, it joins, and then passes the upgrade.
|
| revision-target | No, default `latest` |
- `latest` is the default. When rolling out a new revision to an instance, the latest available revision is chosen.
- `next` trades speed for smaller changes. When rolling out a new revision to an instance, the next available revision is chosen.
The available revisions for an instance are revisions which are not yet deployed, or revisions which have rolled out in previous instances. |
| revision-change | No, default `when-failing` |
- `always` is the most aggressive setting. A new, available revision may always replace the one which is currently rolling out.
- `when-failing` is the default. A new, available revision may replace the one which is currently rolling out if this is failing.
- `when-clear` is the most conservative setting. A new, available revision may never replace one which is currently rolling out.
Revision targets will never automatically change inside [revision block window](#block-change), but may be set by manual intervention at any time. |
| max-risk | No, default `0` | May only be used with `revision-change="when-clear"` and `revision-target="next"`. The maximum amount of _risk_ to roll out per new revision target. The default of `0` results in the next build always being chosen, while a higher value allows skipping intermediate builds, as long as the cumulative risk does not exceed what is configured here. |
| min-risk | No, default `0` | Must be less than or equal to the configured `max-risk`. The minimum amount of _risk_ to start rolling out a new revision. The default of `0` results in a new revision rolling out as soon as anything is ready, while a higher value lets the system wait until enough cumulative risk is available. This can be used to avoid blocking a lengthy deployment process with trivial changes. |
| max-idle-hours | No, default `8` | May only be used when `min-risk` is specified, and greater than `0`. The maximum number of hours to wait for enough cumulative risk to be available, before rolling out a new revision. |
##### test
Meaning depends on where it is located:
| Parent | Description |
| --- | --- |
| `` `` | If present, the application is deployed to the [`test`](../../operations/environments.html#test) environment, and system tested there, even if no prod zones are deployed to. Also, when specified, system tests _must_ be present in the application test package. See guides for [getting to production](../../operations/production-deployment.html).
If present in an `` element, system tests are run for that specific instance before any production deployments of the instance may proceed — otherwise, previous system tests for any instance are acceptable. |
| `` `` `` | If present, production tests are run against the production region with id contained in this element. A test must be _after_ a corresponding [region](#region) element. When specified, production tests _must_ be preset in the application test package. See guides for [getting to production](../../operations/production-deployment.html). |
| Attribute | Mandatory | Values |
| --- | --- | --- |
| cloud-account | No | For [system tests](../../operations/automated-deployments.html#system-tests) only: account to deploy to with [Vespa Cloud Enclave](../../operations/enclave/enclave). Overrides parent's use of cloud-account. Cloud account _must not_ be specified for [production tests](../../operations/automated-deployments.html#production-tests), which always run in the account of the corresponding deployment. |
| empty-host-ttl | No | How long to keep hosts after they become empty. Overrides parent's use of empty-host-ttl. See [empty host TTL](#empty-host-ttl). |
##### staging
In ``, or ``. If present, the application is deployed to the [`staging`](../../operations/environments.html#staging) environment, and tested there, even if no prod zones are deployed to. If present in an `` element, staging tests are run for that specific instance before any production deployments of the instance may proceed — otherwise, previous staging tests for any instance are acceptable. When specified, staging tests _must_ be preset in the application test package. See guides for [getting to production](../../operations/production-deployment.html).
| Attribute | Mandatory | Values |
| --- | --- | --- |
| cloud-account | No | Account to deploy to with [Vespa Cloud Enclave](../../operations/enclave/enclave). Overrides parent's use of cloud-account. |
| empty-host-ttl | No | How long to keep hosts after they become empty. Overrides parent's use of empty-host-ttl. See [empty host TTL](#empty-host-ttl). |
##### prod
In ``, **or** in ``. If present, the application is deployed to the production regions listed inside this element, under the specified instance, after deployments and tests in the `test` and `staging` environments.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| cloud-account | No | Account to deploy to with [Vespa Cloud Enclave](../../operations/enclave/enclave). Overrides parent's use of cloud-account. |
| empty-host-ttl | No | How long to keep hosts after they become empty. Overrides parent's use of empty-host-ttl. See [empty host TTL](#empty-host-ttl). |
##### region
In ``, ``, ``, or ``. The application is deployed to the production [region](../../operations/zones.html) identified by the `name` attribute.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| name | Yes | The region identifier, e.g. `aws-us-east-1c`. See [zones](../../operations/zones.html) for the list of available regions. Mandatory _unless_ the region name is specified in the element body, but this is not compatible with specifying [``](#availability-zone) children. |
| fraction | No | Only when this region is inside a group: The fractional membership in the group. |
| cloud-account | No | Account to deploy to with [Enclave](../../operations/enclave/enclave). Overrides parent's use of cloud-account. |
| empty-host-ttl | No | How long to keep hosts after they become empty. Overrides parent's use of empty-host-ttl. See [empty host TTL](#empty-host-ttl). |
###### availability-zone
In [``](#region) or [`` (dev)](#region-dev). The element body must be one of the availability zone identifiers listed in [zones](../../operations/zones.html). At least one availability zone _must_ be specified when deploying to a region that supports more than one availability zone. The application instance will be spread out evenly across these availability zones for resiliency. See [availability zones](../../operations/az.html) for more.
Example:
```xml
use1-az1
use1-az2
```
##### dev
In ``. Optionally used to control deployment settings for the [dev environment](../../operations/environments.html). This can be used specify a different cloud account, tags, private endpoints, and availability zones per region.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| tags | No | Space-separated tags which can be referenced to make [deployment variants](../../operations/deployment-variants.html). |
| cloud-account | No | Account to deploy to with [Vespa Cloud Enclave](../../operations/enclave/enclave). Overrides parent's use of cloud-account. |
| empty-host-ttl | No | How long to keep hosts after they become empty. Overrides parent's use of empty-host-ttl. See [empty host TTL](#empty-host-ttl). |
##### delay
In ``, ``, ``, ``, or ``. Introduces a delay which must pass after completion of all previous steps, before subsequent steps may proceed. This may be useful to allow some grace time to discover errors before deploying a change in additional zones, or to gather higher-level metrics for a production deployment for a while, before evaluating these in a production test. The maximum total delay for the whole deployment spec is 48 hours. The delay is specified by any combination of the `hours`, `minutes` and `seconds` attributes.
By default, a delay applies to both application revision changes and Vespa platform upgrades. It is possible to restrict the delay to just one kind of change using the `revision` and `version` attributes, with the same semantics as on [block-change](#block-change). When a delay applies only to one kind of change, the other kind passes through the step with no delay.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| hours | No, default `0` | The number of hours to delay. |
| minutes | No, default `0` | The number of minutes to delay. |
| seconds | No, default `0` | The number of seconds to delay. |
| revision | No, default `true` | Set to `false` to let application revision deployments proceed without this delay. |
| version | No, default `true` | Set to `false` to let Vespa platform upgrades proceed without this delay. |
The below example delays Vespa platform upgrades by 1 hour between the surrounding steps, while application revision deployments proceed with no delay:
```xml
```
##### parallel
In ``, ``, or ``. Runs the contained steps in parallel: instances if in ``, or primitive steps (deployments, tests or delays) or a series of these (see [steps](#steps)) otherwise. Multiple `` elements are permitted. The following example will deploy to `us-west-1` first, then to `us-east-3` and `us-central-1` simultaneously, and, finally to `eu-west-1`, once both parallel deployments have completed:
```xml
```
##### steps
In ``. Runs the contained parallel or primitive steps (deployments, tests or delays) serially. The following example will in parallel:
1. deploy to `us-east-3`,
2. deploy to `us-west-1`, then delay 1 hour, and run tests for `us-west-1`, and
3. delay for two hours.
Thus, the parallel block is complete when both deployments are complete, tests are successful for the second deployment, and at least two hours have passed since the block began executing.
```xml
us-west-1
```
##### tester
In ``, `` and ``. Specifies container settings for the tester application container, which is used to run system, staging and production verification tests.
The allowed elements inside this are [``](../applications/services/services.html#nodes).
The tester always runs on a single node, so `count` must be `1`.
```xml
```
##### endpoints (global)
In ``, without any `` declared **or** in ``: This allows _global_ endpoints, via one or more [``](#endpoint-global) elements; and [zone endpoint](#endpoint-zone) and [private endpoint](#endpoint-private) elements for cloud-native private network configuration.
##### region (dev)
In ``. Declares the [availability zones](#availability-zone) to use when deploying to this region in the [dev](../../operations/environments.html#dev) environment. Unlike [``](#region) in ``, this does not cause a deployment to the zone; it only applies when deploying to that dev zone. Only specify this for regions that support more than one availability zone. At least one `` child must be specified, and each region can be declared only once.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| name | Yes | The region identifier, e.g. `gcp-us-central1`. |
Example:
```xml
us-central1-a
```
##### endpoints (dev)
In ``. This allows [zone endpoint](#endpoint-zone) elements for cloud-native private network configuration for [dev](../../operations/environments.html#dev) deployments. Note that [private endpoints](#endpoint-private) are only supported in `prod`.
##### endpoint (global)
In `` or ``. Specifies a global endpoint for this application. Each endpoint will point to the regions that are declared in the endpoint. If no regions are specified, the endpoint defaults to the regions declared in the `` element. The following example creates a default endpoint to all regions, and a _us_ endpoint pointing only to US regions.
```xml
```
| Attribute | Mandatory | Values |
| --- | --- | --- |
| id | No | The identifier for the endpoint. This will be part of the endpoint name that is generated. If not specified, the endpoint will be the default global endpoint for the application. |
| container-id | Yes | The id of the [container cluster](/en/reference/applications/services/container.html) to which requests to the global endpoint is forwarded. |
Global endpoints are implemented using Route 53 and healthchecks, to keep active zones in rotation. See [BCP](#bcp) for advanced configurations.
##### endpoint (zone)
In `` or ``, with `type='zone'`. Used to disable public zone endpoints. _Non-public endpoints can not be used in global endpoints, which require that all constituent endpoints are public._ The example disables the public zone endpoint for the `my-container` container cluster in all regions, except where it is explicitly enabled, in `region-1`. Changing endpoint visibility will make the service unavailable for a short period of time.
```xml
```
| Attribute | Mandatory | Values |
| --- | --- | --- |
| type | Yes | Private endpoints are specified with `type='zone'`. |
| container-id | Yes | The id of the [container cluster](/en/reference/applications/services/container.html) to disable public endpoints for. |
| enabled | No | Whether a public endpoint for this container cluster should be enabled; default `true`. |
##### endpoint (private)
In `` or ``, with `type='private'`. Specifies a private endpoint service for this application. Each service will be launched in the regions that are declared in the endpoint. If no regions are specified, the service is launched in all regions declared in the `` element, that support any of the declared [access types](#allow). The following example creates a private endpoint in two specific regions.
```xml
```
| Attribute | Mandatory | Values |
| --- | --- | --- |
| type | Yes | Private endpoints are specified with `type='private'`. |
| container-id | Yes | The id of the [container cluster](/en/reference/applications/services/container.html) to which requests to the private endpoint service is forwarded. |
| auth-method | No | The authentication method to use with this [private endpoint](/en/operations/private-endpoints.html).
Must be either `mtls` or `token`. Defaults to mTLS if not included. |
##### allow
In ``. Allows a principal identified by the URN to set up a connection to the declared private endpoint service. This element must be repeated for each additional URN. An endpoint service will only consider allowed URNs of a compatible type, and will only be created if at least one compatible access type-and-URN is given:
- For AWS deployments, specify `aws-private-link`, and an _ARN_.
- For GCP deployments, specify `gcp-service-connect`, and a _project ID_
```xml
```
| Attribute | Mandatory | Values |
| --- | --- | --- |
| with | Yes | The private endpoint access type; must be `aws-private-link` or `gcp-service-connect`. |
| arn | Maybe | Must be specified with `aws-private-link`. See [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html) for more details. |
| project | Maybe | Must be specified with `gcp-service-connect`. See [GCP documentation](https://cloud.google.com/vpc/docs/configure-private-service-connect-services) for more details. |
##### bcp
In `` or ``. Defines the BCP (Business Continuity Planning) structure of this instance: Which zones should take over for which others during the outage of a zone and how fast they must have the capacity ready. Autoscaling uses this information to decide the ideal cpu load of a zone. If this element is not defined, it is assumed that all regions covers for an equal share of the traffic of all other regions and must have that capacity ready at all times.
If a bcp element is specified at the root, and explicit instances are used, that bcp element becomes the default for all instances that does not contain a bcp element themselves. If a BCP element contains no group elements it will implicitly define a single group of all the regions of the instance in which it is used.
See [BCP test](https://cloud.vespa.ai/en/reference/bcp-test.html) for a procedure to verify that your BCP configuration is correct.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| deadline | No |
The max time after a region becomes unreachable until the other regions in its BCP group must be able to handle the traffic of it, given as a number of minutes followed by 'm', 'h' or 'd' (for minutes, hours or days). The default deadline is 0: Regions must at all times have capacity to handle BCP traffic immediately.
By providing a deadline, autoscaling can avoid the cost of provisioning additional resources for BCP capacity if it predicts that it can grow to handle the traffic faster than the deadline in a given cluster.
This is the default deadline to be used for all groups that don't specify one themselves.
|
Example:
```xml
```
##### group
In ``. Defines a bcp group: A set of regions whose members cover for each other during a regional outage.
Each region in a group will (as allowed, when autoscaling ranges are configured) provision resources sufficient to handle that any other single region in the group goes down. The traffic of the region is assumed to be rerouted in equal amount to the remaining regions in the group. That is, if a group has one member, no resources will be provisioned to handle an outage in that member. If a group has two members, each will aim to provision sufficient resources to handle the actual traffic of the other. If a group has three members, each will provision to handle half of the traffic observed in the region among the two others which receives the most traffic.
A region may have fractional membership in multiple groups, meaning it will handle just that fraction of the traffic of the remaining members, and vice versa. A regions total membership among groups must always sum to exactly 1.
A group may also define global endpoints for the region members in the group. This is exactly the same as defining the endpoint separately and repeating the regions of the group under the endpoint. Endpoints under a group cannot contain explicit region sub-elements.
| Attribute | Mandatory | Values |
| --- | --- | --- |
| deadline | No |
The deadline of this BCP group. See deadline on the BCP element.
|
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [deployment.xml reference](#page-title)
- [deployment](#deployment)
- [instance](#instance)
- [block-change](#block-change)
- [backup](#backup)
- [resource-tags](#resource-tags)
- [upgrade](#upgrade)
- [test](#test)
- [staging](#staging)
- [prod](#prod)
- [region](#region)
- [availability-zone](#availability-zone)
- [dev](#dev)
- [delay](#delay)
- [parallel](#parallel)
- [steps](#steps)
- [tester](#tester)
- [endpoints (global)](#endpoints-global)
- [region (dev)](#region-dev)
- [endpoints (dev)](#endpoints-dev)
- [endpoint (global)](#endpoint-global)
- [endpoint (zone)](#endpoint-zone)
- [endpoint (private)](#endpoint-private)
- [allow](#allow)
- [bcp](#bcp)
- [group](#group)
---
### services.xml - document-processing
[](/en/reference/applications/services/docproc.html.md "View as Markdown")
#### services.xml - document-processing
[](/en/reference/applications/services/docproc.html.md "View as Markdown")
This is the [document-processing](../../../applications/document-processors.html) reference in [services.xml](services.html):
```
[container](container.html)document-processing [numnodesperclient, preferlocalnode, maxmessagesinqueue, maxqueuebytesize, maxqueuewait, maxconcurrentfactor, documentexpansionfactor, containercorememory][include](container.html#include)[documentprocessor [class, bundle, id, idref, provides, before, after]](#documentprocessor)[provides](#provides)[before](processing.html#before)[after](processing.html#after)[map](#map)[field [doctype, in-document, in-processor]](#map)[chain [name, id, idref, inherits, excludes, documentprocessors]](#chain)[map](#map)[field [doctype, in-document, in-processor]](#map)[inherits](processing.html#inherits)[chain](processing.html#chain)[exclude](processing.html#exclude)[documentprocessor [class, bundle, id, idref, provides, before, after]](#documentprocessor)[provides](#provides)[before](processing.html#before)[after](processing.html#after)[map](#map)[field [doctype, in-document, in-processor]](#map)[phase [id, idref, before, after]](processing.html#phase)[before](processing.html#before)[after](processing.html#after)[threadpool](#threadpool)
```
The root element of the _document-processing_ configuration model.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| numnodesperclient | optional | | |
**Deprecated:** Ignored and deprecated, will be removed in Vespa 9.
Set to some number below the amount of nodes in the cluster to limit how many nodes a single client can connect to. If you have many clients, this can reduce the memory usage on both document-processing and client nodes. |
| preferlocalnode | optional | | false |
**Deprecated:** Ignored and deprecated, will be removed in Vespa 9.
Set to always prefer sending to a document-processing node running on the same host as the client. You should use this if you are running a client on each document-processing node. |
| maxmessagesinqueue | | | |
|
| maxqueuebytesize | | | |
**Deprecated:** Ignored and deprecated, will be removed in Vespa 9.
|
| maxqueuewait | optional | | |
The maximum number of seconds a message should wait in queue before being processed. Docproc will adapt its queue size to adhere to this. If the queue is full, new messages will be replied to with SESSION\_BUSY.
|
| maxconcurrentfactor | | | |
|
| documentexpansionfactor | optional | | |
|
| containercorememory | | | |
|
##### Document Processor elements
_documentprocessor_ elements are contained in [docproc chain elements](#chain) or in the _document-processing_ root.
A documentprocessor element is either a document processor definition or document processor reference. The rest of this section deals with document processor definitions; document processor references are described in [docproc chain elements](#docproc-chain-elements).
A documentprocessor definition causes the creation of exactly one document processor instance. This instance is set up according to the content of the documentprocessor element.
A documentprocessor definition contained in a docproc chain element defines an _inner document processor_. Otherwise, it defines an _outer document processor._
For inner documentprocessors, the name must be unique inside the docproc chain. For outer documentprocessors, the component id must be unique. An inner documentprocessor is not permitted to have the same name as an outer documentprocessor.
Optional sub-elements:
- provides, a single name that should be added to the provides list
- before, a single name that should be added to the before list
- after, a single name that should be added to the after list
- config (one or more)
For more information on provides, before and after, see [Chained components](../../../applications/chaining.html).
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| class | | | | |
| bundle | | | | |
| id | required | | |
The component id of the documentprocessor instance.
|
| idref | | | | |
| provides | optional | | |
A space-separated list of names that represents what this documentprocessor produces.
|
| before | optional | | |
A space-separated list of phase or provided names. Phases or documentprocessors providing these names will be placed later in the docproc chain than this document processor.
|
| after | optional | | |
A space-separated list of phase or provided names. Phases or documentprocessors providing these names will be placed earlier in the docproc chain than this document processor.
|
###### documentprocessor
Defines a documentprocessor instance of a user specified class.
```xml
...
```
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | | | The component id of the documentprocessor instance. |
| class | optional | | | A component specification containing the name of the class to instantiate to create the document processor instance. If missing, copied from id. |
| bundle | optional | | | The bundle containing the class: The name in \ in pom.xml. If a bundle is not specified, the bundle containing document processors bundled with Vespa is used. |
##### Docproc chain elements
Specifies how a docproc chain should be instantiated, and how the contained document processors should be ordered.
###### chain
Contained in _document-processing_. Refer to the [chain reference](processing.html#chain). Chains can [inherit](processing.html#inherits) document processors from other chains and use [phases](processing.html#phase) for ordering. Optional sub-elements:
- [documentprocessor element](#documentprocessor) (one or more), either a documentprocessor reference or documentprocessor definition. If the name given for a documentprocessor matches an _outer documentprocessor_, it is a _documentprocessor reference_ - otherwise, it is a _documentprocessor definition_. If it is a documentprocessor definition, it is also an implicit documentprocessor reference saying: use _exactly_ this documentprocessor. All these documentprocessor elements must have different name.
- [phase](processing.html#phase) (one or more).
- [config](../config-files.html#generic-configuration-in-services-xml) (one or more - will apply to all _inner_ documentprocessors in this docproc chain, unless overridden by individual inner documentprocessors).
##### Map
Set up a field name mapping from the name(s) of field(s) in the input documents to the names used in a deployed docproc. The purpose is to reuse functionality without changing the field names. The example below shows the configuration:
```xml
```
In the example, a chain is deployed with 2 docprocs.
For the chain, a mapping from _key_ to _id_ is set up. Imagine that some or all of the docprocs in the chain read and write to a field called _id_, but we want this functionality to the document field _key_.
Furthermore, a similar thing is done for the `CityDocProc`: The docproc accesses the field _city_, whereas it's called _town_ in the feed. The mapping only applies to the document type _restaurant_.
The `CarDocProc` accesses a field called _cyl_. In this example this is mapped to the field _cylinders_ of a struct _engine_ using a dotted notation.
If you specify mappings on different levels of the config (say both for a cluster and a docproc), the mapping closest to the actual docproc will take precedence.
##### threadpool
Available since Vespa 8.601.12
Specifies configuration for the thread pool used by document processor chains. All values scale with the number of vCPU—see the [container tuning example](../../../performance/container-tuning.html#container-worker-threads-example). When all workers are busy, new document processing requests are rejected.
###### threads
Number of worker threads per vCPU. Default value is `1`. The pool runs with `threads * vCPU` workers.
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [services.xml - document-processing](#page-title)
- [Document Processor elements](#document-processor-elements)
- [documentprocessor](#documentprocessor)
- [Docproc chain elements](#docproc-chain-elements)
- [chain](#chain)
- [Map](#map)
- [threadpool](#threadpool)
- [threads](#threadpool-threads)
---
### hosts.xml
[](/en/reference/applications/hosts.html.md "View as Markdown")
#### hosts.xml
[](/en/reference/applications/hosts.html.md "View as Markdown")
_hosts.xml_ is a configuration file in an [application package](application-packages.html). Elements:
```
hosts[host [name]](#host)[alias](#alias)
```
The purpose of _hosts.xml_ is to add aliases for real hostnames to self-defined aliases. The aliases are used in [services.xml](services/services.html) to map service instances to hosts. It is only needed when deploying to multiple hosts.
##### host
Sub-elements:
- [`alias`](#alias)
Example:
```xml
SEARCH0
CONTAINER0
SEARCH1
CONTAINER1
```
##### alias
Alias used in [services.xml](services/services.html) to refer to the host.
Copyright © 2026 - [Cookie Preferences](#)
---
### services.xml - http
[](/en/reference/applications/services/http.html.md "View as Markdown")
#### services.xml - http
[](/en/reference/applications/services/http.html.md "View as Markdown")
This is the reference for the `http` subelement of [container](container.html) in [services.xml](services.html). The http block is used to configure http servers and filters - when this element is present, the default http server is disabled.
```
http[server [id, port]](#server)[ssl](#ssl)[private-key-file](#private-key-file)[certificate-file](#certificate-file)[ca-certificates-file](#ca-certificates-file)[client-authentication](#client-authentication)[protocols](#protocols)[cipher-suites](#cipher-suites)[ssl-provider [class, bundle]](#ssl-provider)[filtering](#filtering)[filter [id, class, bundle, provides, before, after]](#filter)[provides](#provides)[before](#before)[after](#after)[filter-config](#filter-config)[request-/response-chain [id, inherits, excludes]](#chain)[binding](#binding)[filter [id, class, bundle, provides, before, after]](#filter)[provides](#provides)[before](#before)[after](#after)[filter-config](#filter-config)[inherits](#inherits)[chain](#inheritedchain)[exclude](#exclude)[phase [id, before, after]](#phase)[before](#before)[after](#after)
```
Most elements takes optional [config](../config-files.html#generic-configuration-in-services-xml) elements, see example in [server](#server).
Note: To bind the search handler port (i.e. the handler for queries), refer to [search bindings](search.html#binding).
Example:
```xml
http://*/*
http://*:8080/*
http://*:9000/path
```
##### server
The definition of a http server. Configure the server using [jdisc.http.connector.def](https://github.com/vespa-engine/vespa/blob/master/container-disc/src/main/resources/configdefinitions/jdisc.http.jdisc.http.connector.def).
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The component ID
|
| port | optional | number | The web services port of the [environment variables](/en/operations/self-managed/files-processes-and-ports.html#environment-variables) |
Server port
|
| default-request-chain | optional | string | |
The default request chain to use for unmatched requests
|
| default-response-chain | optional | string | |
The default response chain to use for unmatched requests
|
Example:
```xml
90
```
##### ssl
Setup TLS on HTTP server using credentials provided in PEM format.
##### private-key-file
Path to private key file in PEM format.
##### certificate-file
Path to certificate file in PEM format.
##### ca-certificates-file
Path to CA certificates file in PEM format.
##### client-authentication
Client authentication. Supported values: _disabled_, _want_ or _need_.
##### protocols
Comma-separated list of TLS protocol versions to enable. Example: _TLSv1.2,TLSv1.3_.
##### cipher-suites
Comma-separated list of TLS cipher suites to enable. The specified ciphers must be supported by JDK installation. Example: _TLS\_AES\_256\_GCM\_SHA384,TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_GCM\_SHA384_.
##### ssl-provider
Setup TLS on the HTTP server through a programmatic Java interface. The specified class must implement the [SslProvider](https://javadoc.io/doc/com.yahoo.vespa/container-disc/latest/com/yahoo/jdisc/http/SslProvider.html) interface.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| class | required | string | |
The class name
|
| bundle | required | string | |
The bundle name
|
##### filtering
`filtering` is for configuring http filter chains. Sub-elements:
- [filter](#filter)
- [request-chain](#chain)
- [response-chain](#chain)
Example:
```xml
http://*/
http://*/
```
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| strict-mode | optional | boolean | true |
When set to true, all requests must match a filter. For any requests not matching, an HTTP 403 response is returned.
|
##### binding
Specifies that requests/responses matching the given URI pattern should be sent through the [request-chain/response-chain](#chain).
##### filter
The definition of a single filter, for referencing when defining chains. If a single filter is to be used in different chains, it is cleaner to define it directly under `http` and then refer to it with `id`, than defining it inline separately for each chain. The following filter types are supported:
- RequestFilter
- ResponseFilter
- SecurityRequestFilter
- SecurityResponseFilter
Security[Request/Response]Filters are automatically wrapped in Security[Request/Response]FilterChains. This makes them behave like regular Request/Response filters with respect to chaining.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The component ID
|
| class | optional | string | id |
The class of the component, defaults to id
|
| bundle | optional | string | id or class |
The bundle to load the component from, defaults to class or id (if no class is given)
|
| before | optional | string | |
Space separated list of phases and/or filters which should succeed this phase
|
| class | optional | string | id |
Space separated list of phases and/or filters which should precede this phase
|
Sub-elements:
- [provides](#provides)
- [before](#before)
- [after](#after)
- [filter-config](#filter-config)
Example:
```xml
```
##### provides
A name provided by a filter for phases and other filters to use as dependencies. Contained in [filter](#filter) and [filter](#filter) (in chain).
##### before
The name of a phase or filter which should succeed this phase or filter. `before` tags may be used if it is necessary to define filters or phases which always should succeed this filter or phase in a chain. In other words, the phase or filter defined is placed _before_ name in the tag. Contained in [filter](#filter), [filter](#filter) (in chain) and [phase](#phase).
##### after
The name of a phase or filter which should precede this phase or filter. `after` tags may be used if it is necessary to define filters or phases which always should precede this filter or phase in a chain. In other words, the phase or filter defined is placed _after_ the name in the tag. Contained in [filter](#filter), [filter](#filter) (in chain) and [phase](#phase). Example:
```xml
Authorization
LastFilters
Earlyfilters
```
##### filter-config
Only used to configure filters that are configured with `com.yahoo.jdisc.http.filter.security.FilterConfig`. This is the case for all filters provided in JDisc bundles.
##### request-chain/response-chain
Defines a chain of request filters or response filters, respectively. A chain is a set ordered by dependencies. Dependencies are expressed through phases, which may depend upon other phases, or filters.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| inherits | | string | |
A space separated list of chains this chain should include the contents of
|
| excludes | | string | |
A space separated list of filters (contained in an inherited chain) this chain should not include
|
Sub-elements:
- [binding](#binding)
- [filter](#filter). Refer to or define a filter. _config_ or _filter-config_ can not be added to references, only filter definitions.
- [inherits](#inherits)
- [phase](#phase)
Examples:
```xml
http://*/*
http://*:8080/*
http://*:9000/path
```
##### inherits
Wrapper element for information about which chains, if any, a chain should inherit, and how. Contained in [request-chain](#chain) and [response-chain](#chain). Sub-elements:
- (inherited) [chain](#inheritedchain)
- [exclude](#exclude)
##### (inherited) chain
The ID of a chain which this chain should inherit, i.e. include all filters and phases from. Use multiple `chain` tags if it is necessary to combine the filters from multiple chains. Contained in [inherits](#inherits).
##### exclude
A filter the chain under definition should exclude from the chain or chains it inherits from. Use multiple `exclude` tags to exclude multiple filters. Contained in [inherits](#inherits). Example:
```xml
idOfSomeInheritedChain
idOfUnwantedFilter
idOfYetAnotherUnwantedFilter
```
##### phase
Defines a phase, which is a checkpoint to help order filters. Filters and other phases may depend on a phase to be able to make assumptions about the order of filters. Contained in [chain](#chain).
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The ID, or name, which other phases and filters may depend upon as a [successor](#before) or [predecessor](#after)
|
| before | optional | string | |
Space separated list of phases and/or filters which should succeed this phase
|
| after | optional | string | |
Space separated list of phases and/or filters which should precede this phase
|
Sub-elements:
- [before](#before)
- [after](#after)
Example:
```xml
Authorization
```
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [services.xml - http](#page-title)
- [server](#server)
- [ssl](#ssl)
- [private-key-file](#private-key-file)
- [certificate-file](#certificate-file)
- [ca-certificates-file](#ca-certificates-file)
- [client-authentication](#client-authentication)
- [protocols](#protocols)
- [cipher-suites](#cipher-suites)
- [ssl-provider](#ssl-provider)
- [filtering](#filtering)
- [binding](#binding)
- [filter](#filter)
- [provides](#provides)
- [before](#before)
- [after](#after)
- [filter-config](#filter-config)
- [request-chain/response-chain](#chain)
- [inherits](#inherits)
- [(inherited) chain](#inheritedchain)
- [exclude](#exclude)
- [phase](#phase)
---
### services.xml - processing
[](/en/reference/applications/services/processing.html.md "View as Markdown")
#### services.xml - processing
[](/en/reference/applications/services/processing.html.md "View as Markdown")
This document explains the syntax and semantics of the `processing` subelement of the [`container`](container.html) part of `services.xml`. `processing` is for configuring a pure [request-response processing](../../../applications/processing.html) application, with no particular dependency to search or query handling. The processing block is used to configure [processors](../../../applications/processing.html):
```
processing[binding](#binding)[processor [id, class, bundle, provides, before, after]](#processor)[provides](#provides)[before](#before)[after](#after)[renderer [id, class, bundle]](#renderer)[chain [id, inherits, excludes]](#chain)[processor [idref, id, class, bundle, provides, before, after]](#processor-in-chain)[provides](#provides)[before](#before)[after](#after)[inherits](#inherits)[chain](#chain)[exclude](#exclude)[phase [id, before, after]](#phase)[before](#before)[after](#after)
```
Example:
```xml
```
##### binding
The URI to map the ProcessingHandler to. The default binding is `http://*/processing/*`. Multiple elements are allowed. Example:
```xml
http://*/processing/*
```
##### processor
The definition of a single processor, for referencing when defining chains. If a single processor is to be used in multiple chains, it is cleaner to define it directly under `processing` and then refer to it with `idref`, than defining it inline separately for each chain.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The component ID
|
| class | optional | string | |
The class of the component, defaults to id
|
| bundle | optional | string | |
The bundle to load the component from, defaults to class or id (if no class is given)
|
| before | optional | string | |
Space separated list of phases and/or processors which should succeed this processor
|
| after | optional | string | |
Space separated list of phases and/or processors which should precede this processor
|
Example:
```xml
```
##### renderer
The definition of a renderer, for use by a Handler.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The component ID
|
| class | optional | string | |
The class of the component, defaults to id
|
| bundle | optional | string | |
The bundle to load the component from, defaults to class or id (if no class is given)
|
Example:
```xml
```
##### processor (in chain)
Reference to or inline definition of a processor in a chain. If inlining, same as [processor](#processor) - if referring to, use _idref_ attribute:
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| idref | | string | |
Reference to the definition of this processor.
|
Example:
```xml
```
##### provides
A name provided by a processor for phases and other processors to use as dependencies. Example:
```xml
IntentAnalysis
```
##### chain
An element for defining a chain of [processors](processing.html) / [searchers](search.html) / [document processors](docproc.html) (i.e. components). A chain is a set ordered by dependencies. Dependencies are expressed through phases, which may depend upon other phases, or components. For an incoming request from the network, the chain named _default_ will be invoked. Refer to [Chained components](../../../applications/chaining.html) for a guide. Requires one of _id_ or _idref_.
Searcher, Document processing and Processing chains can be modified at runtime without restarts. Modification includes adding/removing processors in chains and changing names of chains and processors. Make the change and [deploy](/en/basics/applications.html#deploying-applications). Some changes require a container restart, refer to [reconfiguring document processing](/en/applications/document-processors.html#reconfiguring-document-processing).
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| idref | | string | |
A reference to a defined chain. Mutually exclusive with _id_. If _idref_ is used, no other attributes apply.
|
| id | required | string | |
The chain ID. Required unless _idref_ is used
|
| inherits | optional | string | |
A space-separated list of chains this chain should include the contents of - see example below.
|
| excludes | optional | string | |
A space-separated list of processors (contained in an inherited chain) this chain should not include. The exclusion is done before any consolidation of component references when inheriting chains. Example:
```xml
```
|
| class | optional | string | |
|
| name | | | |
|
| documentprocessors | | | |
|
##### inherits
Inherit from one or more parent chain(s).
When a search chain inherits from another search chain, it subsumes the phases and the _searcher references_ (both implicitly and explicitly defined) from the parent chain.
If two or more inherited component references have the same name, a new component specification matching those will be used instead. If that is not possible, an error will be signaled (i.e. if the version specifications can not be consolidated or if they require component definitions from different chains).
The component references determines which instances are used in the resulting chain instance.
A component reference is a component specification that says: there shall be exactly one component in this chain with the given name, and this component must match the version specification.
A component reference _overrides_ any inherited component references with the same name (i.e. the inherited ones are ignored).
If several components match a given component reference, the newest (as determined by the version) is used.
##### exclude
Exclude components from inherited chains.
##### phase
Defines a phase, which is a named checkpoint to help order components inside a chain. Components and other phases may depend on a phase to be able to make assumptions about the order of components. Refer to the [Chained components](../../../applications/chaining.html) guide.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The ID, or name, which other phases and processors may depend upon as a [successor](#before) or [predecessor](#after).
|
| before | optional | string | |
Space-separated list of phases and/or processors which should succeed this phase
|
| after | optional | string | |
Space-separated list of phases and/or processors which should precede this phase
|
Optional sub-elements:
- before: same as the attribute
- after: same as the attribute
These are equivalent:
```xml
phaseName1
phaseName2
```
Example:
```xml
IntentAnalysis
OtherAnalysis
```
##### before
The name of a phase or component which should succeed this phase or component. Multiple `before` elements can be used to define multiple components or phases which always should succeed this component or phase in a chain. In other words, the phase or component defined is placed _before_ name in the element.
##### after
The name of a phase or component which should precede this phase or component. Multiple `after` elements can be used to define multiple component or phases which always should precede this component or phase in a chain. In other words, the phase or component defined is placed _after_ the name in the element.
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [services.xml - processing](#page-title)
- [binding](#binding)
- [processor](#processor)
- [renderer](#renderer)
- [processor (in chain)](#processor-in-chain)
- [provides](#provides)
- [chain](#chain)
- [inherits](#inherits)
- [exclude](#exclude)
- [phase](#phase)
- [before](#before)
- [after](#after)
---
### services.xml - 'search'
[](/en/reference/applications/services/search.html.md "View as Markdown")
#### services.xml - 'search'
[](/en/reference/applications/services/search.html.md "View as Markdown")
This is the reference for the _search_ part of the container config. Related: [Chained components](../../../applications/chaining.html) and the [federation tutorial](../../../querying/federation.html). The root element of the search configuration, declared as a subelement to [container](container.html):
```
search[binding](#binding)[searcher [id, class, bundle, provides, before, after]](#searcher)[federation [id]](#federation)[source [idref]](#source-reference)[federationoptions [timeout, requestTimeout, optional]](#federationoptions)[source-set [inherits]](#source-set)[target-selector](#target-selector)[chain [id, inherits, excludes]](#chain)[searcher [id, class, bundle, provides, before, after]](#searcher)[federation [id]](#federation)[source [idref]](#source-reference)[federationoptions [timeout, requestTimeout, optional]](#federationoptions)[source-set [inherits]](#source-set)[target-selector](#target-selector)[provider [id, type, cluster, excludes]](#provider)[federationoptions [timeout, requestTimeout, optional]](#federationoptions)[source [id]](#source)[searcher [id, class, bundle, provides, before, after]](#searcher)[renderer [id, class, bundle]](#renderer)[significance](#significance)[threadpool](#threadpool)[threads [max]](#threadpool-threads)[queue](#threadpool-queue)
```
[config](../config-files.html#generic-configuration-in-services-xml) applies to all searchers in the JDisc cluster's search chains, unless overridden by individual search chains or searchers.
##### binding
The URI to map the SearchHandler to. The default binding is `http://*/search/*`. Multiple elements are allowed. Example:
```xml
http://*/search/*
```
##### searcher
Searcher elements are contained in [chain](#chain) elements or in the search root.
A searcher element is either a _definition_ (using _id_) or a _reference_ (using _idref_).
A searcher definition causes the creation of exactly one searcher instance. This instance is set up according to the content of the searcher element. A searcher definition contained in a search chain element defines an _inner searcher_. Otherwise, it defines an _outer searcher._
Searcher definition:
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The component id of the searcher instance. For inner searchers, the id must be unique inside the search chain. For outer searchers, the id must be unique. An inner searcher is not permitted to have the same id as an outer searcher.
|
| class | optional | | |
A component specification containing the name of the class to instantiate to create the searcher instance. If missing, copied from id
|
| bundle | optional | | |
A component specification containing the bundle symbolic name and version used to select the bundle: The name in \ in pom.xml. The class is loaded from this bundle. If no bundle is specified, it defaults to the bundle containing the searchers bundled with Vespa.
|
| provides | optional | | |
A space-separated list of names that represents what this searcher produces. For more information on provides, before and after, see [chained components](../../../applications/chaining.html)
|
| before | optional | | |
A space-separated list of phase or provided names. Phases or searchers providing these names will be placed later in the search chain than this searcher
|
| after | optional | | |
A space-separated list of phase or provided names. Phases or searchers providing these names will be placed earlier in the search chain than this searcher
|
Example:
```xml
```
Searcher reference:
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| idref | required | string | |
Reference to a searcher definition
|
Example:
```xml
```
##### federation
A federation is a [searcher](../../../applications/searchers.html#searchers) - see above. This element implements _federation_ - it defines a searcher instance that sends each query to a set of search chains in parallel and combines the results. Read the [federation guide](../../../querying/federation.html) to learn more and find examples for use of federation and its children [source](#source), [source-set](#source-set) and [target-selector](#target-selector), as well as [provider](#provider).
```xml
```
##### target-selector
Specifies a component that should be used to select search chains to federate to. This component must inherit from com.yahoo.search.federation.selection.TargetSelector. See [component](container.html#component) for attributes and subelements.
##### source-set
Used to duplicate the sources of e.g. the built-in federation searcher:
```xml
…
```
##### source
Reference to a source that should be used by the enclosing federation searcher. Child element [federationoptions](#federationoptions) is optional.
```xml
```
##### federationoptions
Contained in [source](#source) or [provider](#provider). Specifies _how_ a federation searcher should federate to a given search chain. If a federation options A _overrides_ another federation options B, the result is a new federation options containing:
- all the options in B not present in A
- all the options in A
When federating to a source or provider, the federation searcher per default uses the federation options from the search chain. If a [source reference](#source-reference) contains federation options, it overrides the options of the search chain when used from the enclosing federation searcher.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| timeout | optional | number | |
The minimum number of seconds or milliseconds (if ms is present) the federation searcher waits for the federated search chain executing the query
|
| requestTimeout | optional | number | |
The minimum number of seconds or milliseconds (if ms is present) the search chain executing the query should continue execution. In some cases it is useful to set this higher than the timeout, such that a chain can keep waiting for requested data longer than the query is waiting for the chain. This allows queries to populate caches within the search chain even though populating the caches requires waiting longer than the query timeout
|
| optional | optional | true/false | false |
Determines if the federation searcher should wait for this search chain at all. Normally, it only waits for mandatory (i.e. not optional) search chains, and when they are done, cancels the remaining search chains that are not finished. If all the search chains federated to are optional, all of them will be treated as mandatory. All search chains are per default mandatory
|
Example:
```xml
```
##### renderer
The definition of a [search result renderer](../../../applications/result-renderers.html).
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
The component ID
|
| class | optional | string | |
The class of the component, defaults to id
|
| bundle | optional | string | |
The bundle to load the component from: The name in \ in your pom.xml. If no bundle is given, the bundle containing renderers provided by Vespa is used.
|
Example:
```xml
```
##### significance
Contained in [searcher](#searcher). Specifies one or more global significance [models](#model).
```xml
```
The models are either provided by _Vespa_ or generated with [vespa-significance tool](../../operations/self-managed/tools.html#vespa-significance). The order determines model precedence - with the last element having the highest priority. To use these models, schema needs to [enable significance models in the rank-profile](../../schemas/schemas.html#significance).
Sub-elements:
- [model](#model) (required, one or more)
##### model
Contained in [significance](#significance). Specifies [global significance model](../../../ranking/significance#global-significance-model). Models are identified by `model-id` or by providing `url` or `path` to a model file in the application package.
Models with `model-id` are provided by _Vespa_ and listed [here](../../../rag/model-hub.html#significance-models). Example with `model-id`:
```xml
```
A model specified with `url` and `path` are JSON files, which can be also compressed with [zstandard](https://github.com/meta/zstd). Model files can be generated using [vespa-significance tool](../../operations/self-managed/tools.html#vespa-significance). Example with `url`:
```xml
```
Models with `path` should be placed in the application package. The path is relative to the application package root. Example with `path`:
```xml
```
##### chain
Specifies how a search chain should be instantiated, and how the contained searchers should be ordered. Refer to the [chain reference](processing.html#chain) for attributes and child elements. Chains can [inherit](processing.html#inherits) searchers from other chains and use [phases](processing.html#phase) for ordering. Note that [provider](#provider) and [source](#source) elements are also chains. Specify a search chain in a query using [searchChain](../../api/query.html#searchchain).
Example which inherits from the built-in _vespa_ chain so that the searcher can dispatch queries to the content clusters:
```xml
A searcher for ...
```
Optional sub-elements:
- searcher or federation (one or more), either a reference or definition. If the name given for a searcher matches an _outer searcher_, it is a _searcher reference_. Otherwise, it is a _searcher definition_. If it is a searcher definition, it is also an implicit searcher reference saying: use _exactly_ this searcher. All these searcher elements must have different name.
- [phase](processing.html#phase) (one or more).
- [config](../config-files.html#generic-configuration-in-services-xml) (one or more - will apply to all _inner_ searchers in this search chain, unless overridden by individual inner searchers).
You can put search config in separate files in a directory under the application package using [include](container.html#include). Each file must contain one `` element like above. Vespa behaves as if each chain configured within was "inline" in `services.xml`. This is handy when multiple developers need to deploy individual search chains, say in different packages.
**Note:** If using multiple container clusters, the modular search chains will be available in all the clusters.
Each searcher reference must match the _type_ of the searcher definition. So for example the searcher reference _federation idref="myId"_ must match an outer searcher defined as _federation id="myId"_, not _searcher id="myId"_.
##### provider
A provider is a search chain responsible for talking to an external service. Everything covered in [chain](#chain) is also valid for providers.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| id | required | string | |
ID
|
| excludes | optional | | |
|
| type | optional | local | |
Determines which searchers are implicitly added to this search chain to talk to the external service.
|
###### local provider
Local providers are providers with the type set to _local_, accessing a local Vespa cluster (i.e. a content cluster in the same application).
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| cluster | required | string | |
The name of the local cluster.
|
```xml
```
##### source
Defines a source search chain and an associated source.
```xml
providerA
```
The component id specified is the id of the associated source. The associated source consists of all the source search chains with the same source name.
Only a single source search chain can specify the source name using the "id" attribute. This search chain is called the _leader_. The other source search chains must specify the source name using the "idref" attribute. The latter search chains are called participants.
A source can be used for _federation_. When federating to a source, the leader search chain is normally used. To use one of the participant search chains, the following query parameter must be set: source._sourceId_.provider._providerId_.
The id of the source search chain is _sourceId@providerId_. This search chain automatically inherits from the enclosing provider. It also automatically inherits the federation options of the enclosing provider. If the source contains federation options, they override the inherited ones. In all other respects, this search chain behaves like any other search chain.
##### threadpool
Specifies configuration for the thread pool for the jdisc search handler. All parameters are relative to the number of CPU cores—see the [container tuning example](../../../performance/container-tuning.html#container-worker-threads-example). This thread pool also supports the optional `max` attribute, which lets the pool grow up to `max * vCPU` threads under load before shrinking back after 5 seconds of idleness. Requests are rejected once the allowed number of threads is reached, all are busy, and the queue is full.
Optional sub-elements:
- [threads](#threadpool-threads)
- [queue](#threadpool-queue)
###### threads
The number of permanent threads relative to number of vCPU cores. Default value is `10`.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| max | optional | number | equal to `` |
The maximum number of threads relative to vCPU cores. Value must be greater than or equal to ``.
|
###### queue
The size of the request queue relative to effective number of threads. Specify `0` to disable queuing. Default value is `40`.
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [services.xml - 'search'](#page-title)
- [binding](#binding)
- [searcher](#searcher)
- [federation](#federation)
- [target-selector](#target-selector)
- [source-set](#source-set)
- [source](#source-reference)
- [federationoptions](#federationoptions)
- [renderer](#renderer)
- [significance](#significance)
- [model](#model)
- [chain](#chain)
- [provider](#provider)
- [local provider](#local-provider)
- [source](#source)
- [threadpool](#threadpool)
- [threads](#threadpool-threads)
- [queue](#threadpool-queue)
---
### services.xml
[](/en/reference/applications/services/services.html.md "View as Markdown")
#### services.xml
[](/en/reference/applications/services/services.html.md "View as Markdown")
_services.xml_ specifies the clusters an application should have and their capabilities. It is placed in the root of the [application package](../../../basics/applications.html).
Elements:
```
[services [version]](#services)[container [version]](container.html)- specifies a container cluster[content [version]](content.html)- specifies a content cluster[admin [version]](admin.html)- control plane configuration (rarely needed)[routing [version] - how content should be routed (rarely needed)](../../../writing/document-routing.html#routing-services)
```
##### \
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| version | required | number | | 1.0 in this version of Vespa |
Optional subelements (one or more of _container_ or _content_ is required):
- [\](container.html)
- [\](content.html)
- [\](admin.html)
- [\](../../../writing/document-routing.html#routing-services)
The rest of this document describes tags that are used within multiple services tags.
##### \
The _nodes_ element configures the hardware resources of a cluster, and so is used in both container and content clusters. This tag works differently on Vespa Cloud and self-managed instances:
- Vespa Cloud: The number of nodes are specified by a _count_ attribute, and the resources of each node by a [resource](#resources) child element.
- Self-managed: _nodes_ have a [node](#node) child element for each node, A node referred to in _services.xml_ must be defined in [hosts.xml](../hosts.html) using _hostalias_.
It is possible to specify both to make an application package work in both environments, and it is always possible to deploy either type for development on the other: When the nodes tag has Vespa Cloud content it is interpreted as a single-node cluster in a self-hosted environment and vice versa.
| Attribute | type | Default | Description |
| --- | --- | --- | --- |
| **count** | integer or range | | Vespa Cloud: The number of nodes of the cluster. |
| **exclusive** | boolean | false | Optional. Vespa Cloud: If true these nodes will never be placed on shared hosts even when this would otherwise be allowed (which is only for content nodes in some environments). When nodes are allocated exclusively, the resources must match the resources of the host exactly. |
| **groups** | integer or range | | Vespa Cloud content nodes only, optional: Integer or range. Sets the number of groups into which content nodes should be divided. Each group will have an equal share of the nodes, and one or more complete copies of the corpus and index, and each query will be routed to just one group - see [grouped distribution](/en/content/elasticity.html#grouped-distribution). This allows scaling to a higher query load than is possible with just a single group. |
| **group-size** | integer or range | | Vespa Cloud content nodes only, optional: Integer or range where either value can be skipped (replaced by an empty string) to create a one-sided limit. This can be set as an alternative to explicitly setting `groups`: The group sizes used will always be within these limits (inclusive), for any `count`. |
If neither _groups_ nor _group-size_ is set, all nodes belong to a single group. Read more in [topology](../../../performance/topology-and-resizing).
Ranges are expressed by the syntax `[lower-limit, upper-limit]`; Both limits are inclusive. Any value set as a range will be [autoscaled](../../../operations/autoscaling.html).
##### \
Under [nodes](#nodes) on Vespa Cloud: Specifies the resources each node in the cluster should have.
The resources must match a node flavor in [AWS](https://cloud.vespa.ai/en/reference/aws-flavors.html), [GCP](https://cloud.vespa.ai/en/reference/gcp-flavors.html)[Azure](https://cloud.vespa.ai/en/reference/azure-flavors.html), depending on where you are deploying. Exception: If you use remote disk, you can specify any number lower than the max size.
**Subelements:** [\](#gpu)
| Attribute | type | Default | Description |
| --- | --- | --- | --- |
| **vcpu** | float or range | 2 | CPU (virtual threads) |
| **memory** | float or range, each followed by a byte unit, such as "Gb" | 8 Gb in container clusters, 16 Gb in content clusters | Memory |
| **disk** | float or range, each followed by a byte unit, such as "Gb" | 50 in container clusters, 300 in content clusters | Disk space. To fit core dumps/heap dumps, the disk space should be larger than 3 x memory size for content nodes, 2 x memory size for container nodes. If disk size is not explicitly specified, Vespa Cloud chooses a default disk size. The default may be automatically increased to satisfy the minimum disk-to-memory ratio. When both disk and memory are explicitly specified, Vespa Cloud enforces the same minimum ratios. |
| **storage-type** | string (enum) | `any` | The type of storage to use. This is useful to specify local storage when network storage provides insufficient io operations or too noisy io performance:
- `local`: Node-local storage is required.
- `remote`: Network storage must be used.
- `any`: Both remote or local storage may be used.
|
| **disk-speed** | string (enum) | `fast` | The required disk speed category:
- `fast`: SSD-like disk speed is required
- `slow`: This is sized for spinning disk speed
- `any`: Performance does not depend on disk speed (often suitable for container clusters).
|
| **architecture** | string (enum) | `any` | Node CPU architecture:
- `x86_64`
- `arm64`
- `any`: Use any of the available architectures.
|
| **max-cost-factor** | float \>= 1.0 | `1.0` |
Used to allow provisioning of machine types larger than what's specified by these resources when no exactly matching types are currently available for provisioning. This number specifies the max oversize of a provisioned host measured in list price. For example, if this is set to 2, up to twice as costly machines as required may be provisioned.
Exact matches are always preferred when available, and the system will migrate to exactly matching resources in the background as they become available. This has no impact on the resources available to the container node of the Vespa cluster, which will always match the resource spec. This can only be set on [enclave](../../../operations/enclave/enclave) deployed applications.
|
Ranges are expressed by the syntax `[lower-limit, upper-limit]`; Both limits are inclusive. Any value set as a range will be [autoscaled](../../../operations/autoscaling.html).
##### \
Under [nodes](#nodes) on self-managed systems: Specifies a node that should be a member in the cluster.
| Attribute | Required | Value | Default | Description |
| --- | --- | --- | --- | --- |
| hostalias | required | string | |
a host name which must be mapped to a full hostname in [hosts.xml](../hosts.html)
|
##### \
Under [resources](#resources) on Vespa Cloud: Declares GPU resources to provision.
Limitations:
- Available in AWS and GCP zones only
- Valid for container clusters only
| Attribute | type | Description |
| --- | --- | --- |
| **count** | integer | Number of GPUs |
| **memory** | integer, followed by a byte unit, such as "Gb" | Amount of memory per GPU. Total amount of GPU memory available is this number multiplied by `count`. |
Example:
```xml
```
##### Zone-specific configuration
Use _deployment variants_ to express configuration like:
```xml
```
Refer to [deployment variants](../../../operations/deployment-variants.html) for details and more options.
##### Generic configuration using \
Most elements in _services.xml_ accept a sub-element named _config_. _config_ elements can be included on different levels in the XML structure and the lower-level ones will override values in the higher-level ones (example below). The _config_ element must include the attribute _name_, which gives the full name of the configuration option in question, including the namespace. The name can either refer to configuration definitions that are shipped with Vespa or ones that are part of the [application package](../config-files.html). For a complete example on generic configuration see the [application package](../config-files.html#generic-configuration-in-services-xml) reference.
```xml
configured string
```
##### Modular Configuration
Some features are configurable using XML files in subdirectories of the application package. This means that the configuration found in these XML files will be used as if it was inlined in _services.xml_. This is supported for [search chains](search.html#chain), [docproc chains](docproc.html) and [routing tables](../../../writing/document-routing.html#routing-services).
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [services.xml](#page-title)
- [\](#services)
- [\](#nodes)
- [\](#resources)
- [\](#node)
- [\](#gpu)
- [Zone-specific configuration](#zone-specific-configuration)
- [Generic configuration using \](#generic-config)
- [Modular Configuration](#modular)
---
### Testing with Java JUnit tests
[](/en/reference/applications/testing-java.html.md "View as Markdown")
#### Testing with Java JUnit tests
[](/en/reference/applications/testing-java.html.md "View as Markdown")
This is the Vespa Testing reference for [Vespa application system tests](../../applications/testing.html) written in Java, as JUnit 5 unit test.
These tests verify the behaviour of a Vespa application by using its HTTP interfaces. To write tests without Java, see [basic HTTP test reference](testing.html).
See the [testing guide](../../applications/testing.html) for examples of how to run the tests.
##### Test suites
The [testing documentation](../../applications/testing.html) defines three test scenarios, comprised of four test code categories. The _system test framework_ in [com.yahoo.vespa:tenant-cd-api](https://search.maven.org/artifact/com.yahoo.vespa/tenant-cd-api) uses Java annotations to declare what category a JUnit test class belongs to. To run tests with Maven belonging to a specific category, a JUnit 5 test _tag_ must be specified:
```sh
$ mvn test -D test.categories=system -D vespa.test.config=/path-to/test-config.json
```
| Category | Annotation | JUnit tag | Description |
| --- | --- | --- | --- |
| System test | @SystemTest | system | Independent, functional tests |
| Staging setup | @StagingSetup | staging-setup | Set state before upgrade |
| Staging test | @StagingTest | staging | Verify state after upgrade |
| Production test | @ProductionTest | production | Verify domain specific metrics (deprecated, see below) |
For an example including system and staging tests, check out the [sample application test suite](https://github.com/vespa-cloud/examples/tree/main/CI-CD/production-deployment-with-tests-java).
New applications should not write production tests in Java: use the [declarative production test](testing-production.html) format, based on YAML or JSON, instead. The `@ProductionTest` category above is kept only for backward compatibility with applications that already use it — if a test package includes both a declarative production test and a Java `@ProductionTest` bundle, the declarative test is used, and the Java one is ignored.
##### TestNG
Combining Vespa JUnit 5 test suites with unit tests in TestNG is possible. You'll need to explicitly configure Maven's surefire plugin to enable integration for both frameworks. To execute the Vespa test suites specify `-D test.categories=[tag]`, where _[tag]_ is one of the values listed in [Test suites](#test-suites).
```xml
org.apache.maven.plugins
maven-surefire-plugin
org.apache.maven.surefire
surefire-junit-platform
${surefire.vespa.tenant.version}
org.apache.maven.surefire
surefire-testng
${surefire.vespa.tenant.version}
```
Copyright © 2026 - [Cookie Preferences](#)
---
### Production tests
[](/en/reference/applications/testing-production.html.md "View as Markdown")
#### Production tests
[](/en/reference/applications/testing-production.html.md "View as Markdown")
This is the Vespa Testing reference for [production tests](../../applications/testing.html).
Production tests verify domain specific metrics for a production deployment, such as query latency or error rate. Unlike [system](testing.html) and [staging](testing-java.html) tests, production tests check a named [metric preset](#metric-presets) against expected bounds, using Grafana metrics for the application deployed to that zone.
A production test is placed after a deployment zone in [deployment.xml](deployment.html). It acts as a gate for the rollout: if it fails, the rollout stops and subsequent zones will not receive the new version. This is useful in multi-zone deployments where the first zone serves as a canary, but a production test can validate _any_ zone before the rollout proceeds to the next, as described in [production tests](../../operations/automated-deployments.html#production-tests).
##### Production test setup
To use production tests, first deploy to a zone, use [\](deployment.html#delay) for some time, and then run the production tests against that zone. This pattern can be repeated for each zone in the pipeline, to validate it before the rollout proceeds to the next. In the following example, the zone is deployed, then the delay waits 10 minutes, and lastly the production test waits for an additional 10 minutes before aggregating the last 10 minutes of metrics data:
```xml
aws-us-east-1c
aws-us-east-1c
```
The declared production test:
```yaml
name: cpu check
metric: cpu-utilization-container
duration: 10m
max: 85
```
**Note:** Tests based on traffic metrics, such as query or feed latency, need real queries or feeding against the zone during the test window, since without traffic there are no metrics to evaluate and the test will fail.
##### Test suites
Production tests are placed under `tests/production-test/` in the application package, as one or more `.yaml` or `.json` files:
```
$ ls tests/production-test/
tests/production-test/metrics-test.yaml
```
Each file describes one or more tests. A file may contain:
- a single test, as one YAML or JSON object; or
- multiple tests, as a YAML or JSON object with a `tests` field containing a list of test objects.
##### Test file structure
Each test checks a named [metric preset](#metric-presets), over a `duration` window, against a `min` and/or `max` bound. A simple example:
```yaml
name: cpu check
metric: cpu-utilization-container
duration: 5m
max: 85
```
This test fails the production test step if the average CPU utilization of the container cluster, over the last 5 minutes, exceeds 85%.
To place multiple tests in one file, wrap them in a top-level `tests` field, containing a list of test objects. This works the same way for both YAML and JSON:
```yaml
tests:
- name: container cpu check
metric: cpu-utilization-container
duration: 5m
max: 35
- name: content cpu check
metric: cpu-utilization-content
duration: 5m
max: 85
```
###### Test specification
A full list of fields, with description:
| Name | Type | Default | Description |
| --- | --- | --- | --- |
| name | string | | Required. Name used for display purposes in logs and failure messages. Must be unique. |
| metric | string | | Required. The name of one of the [metric presets](#metric-presets). |
| duration | number + granularity suffix | | Required. The duration of the metrics aggregation. Suffix is one of `s|m|h|d`. For example, `duration: 10m` means: first wait 10 minutes, then aggregate the last 10 minutes of data. Tests can have different durations. Since tests run concurrently, the total waiting time is determined by the highest `duration` value, not the sum. |
| min | number | | The inclusive lower bound the query result must satisfy. At least one of `min` and `max` is required. The unit depends on the [metric preset](#metric-presets) used, see its Unit column. |
| max | number | | The inclusive upper bound the query result must satisfy. At least one of `min` and `max` is required. The unit depends on the [metric preset](#metric-presets) used, see its Unit column. |
| instances | list of strings | All instances | Optional. The [instances](deployment.html#instance) the test applies to. The test is skipped for instances not in the list. |
| clusters | list of strings | All clusters | Optional. The cluster ids to check the metric for. The test is run once for each listed cluster, instead of once for the whole application. |
Example of a test which only runs for the `default` instance, and checks the `search` and `feed` container clusters separately:
```yaml
name: container cpu check
metric: cpu-utilization-container
duration: 5m
max: 85
instances: [default]
clusters: [search, feed]
```
A test that is outside its `min`/`max` bounds fails the production test step immediately — production tests are not retried.
##### Examples
A single test, in YAML:
```yaml
name: error rate check
metric: error-rate-pct
duration: 5m
max: 1
```
Multiple tests in a single YAML file:
```yaml
tests:
- name: node count check
metric: node-count-max
duration: 5m
max: 10
- name: documents count check
metric: documents-count-active
duration: 5m
min: 1
```
A single test, in JSON:
```json
{
"name": "query latency check",
"metric": "query-latency-p95",
"duration": "5m",
"max": 200
}
```
Multiple tests in a single JSON file:
```json
{
"tests": [
{
"name": "container feed latency check",
"metric": "container-feed-latency-max",
"duration": "5m",
"max": 100
},
{
"name": "content commit latency check",
"metric": "content-commit-latency-avg",
"duration": "5m",
"max": 50
}
]
}
```
##### Metric presets
Every test checks one of the following named presets, in its `metric` field. Each preset queries the same metrics that back the application's overview dashboard, reduced to a single number. Presets are grouped below by the dashboard tab they come from.
The **Console name** column is a direct match to the graph title shown in that tab in the [Vespa Console](https://console.vespa-cloud.com/)'s metrics view. Use it to find the preset for a graph you're looking at there. The **Metric** column is the exact value to use in the `metric` field, e.g. `metric: query-latency-p95`.
###### Overview
| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Buckets Out of Sync (%) | `buckets-out-of-sync` | percentage (0-100) | Percentage of buckets currently out of sync, fleet-wide. |
| Container: % Nodes Down | `container-nodes-down` | percentage (0-100) | Percentage of active container nodes with at least one service down, fleet-wide. |
| Container Thread Saturation — document-api only | `container-thread-saturation-document-api-only` | percentage (0-100) | Worst-case container thread-pool utilization, fleet-wide, restricted to deployments running document-api only (no search). |
| Container Thread Saturation — search + document-api | `container-thread-saturation-search-document-api` | percentage (0-100) | Worst-case container thread-pool utilization, fleet-wide, restricted to deployments running both search and document-api. |
| Container Thread Saturation — search only | `container-thread-saturation-search-only` | percentage (0-100) | Worst-case container thread-pool utilization, fleet-wide, restricted to deployments running search only (no document-api). |
| Content Executor Saturation | `content-executor-saturation` | percentage (0-100) | Worst-case content-node executor utilization/saturation across the match, docsum, and field-writer executors, fleet-wide. |
| Content: Groups/Nodes Down | `content-groups-nodes-down` | count | Fleet-wide count of cluster groups currently reporting at least one down node. |
| Core Dumps (1h) | `core-dumps` | count | Total number of core dumps processed across content clusters during the time window. |
| Disk Utilization (max per cluster) | `disk-utilization-container` | percentage (0-100) | Maximum disk utilization across nodes in container clusters (logs, working files). |
| Disk Utilization (max per cluster) | `disk-utilization-content` | percentage (0-100) | Maximum disk utilization across nodes in content clusters. |
| Documents per Content Cluster (max) | `documents-per-content-cluster-max` | count | Maximum number of unique documents stored, aggregated across content clusters. |
| Feed Blocked | `feed-blocked` | count | Total number of nodes currently above the feed-block resource limit, across all content clusters. |
| Feed Latency (max) | `feed-latency-max` | milliseconds | Maximum feed latency observed (OK status only). |
| Headroom to Feed Block (per content cluster) | `headroom-to-feed-block-per-content-cluster` | percentage (0-100) | Smallest remaining headroom before the feed-block limit, across memory and disk, fleet-wide. |
| HTTP 2xx Responses (sum) | `http-2xx-responses-sum` | requests/second | Rate of successful (2xx) HTTP responses across the container cluster. |
| HTTP 4xx Responses (sum) | `http-4xx-responses-sum` | requests/second | Rate of client error (4xx) HTTP responses across the container cluster. |
| HTTP 5xx Responses (sum) | `http-5xx-responses-sum` | requests/second | Rate of server error (5xx) HTTP responses across the container cluster. |
| HTTP Read Latency (avg) | `http-read-latency-avg` | milliseconds | Average HTTP-level read latency, including time to receive the request and send the response. |
| HTTP Read Latency (max) | `http-read-latency-max` | milliseconds | Maximum HTTP-level read latency observed. |
| HTTP Read Latency (p95) | `http-read-latency-p95` | milliseconds | 95th percentile HTTP-level read latency. |
| HTTP Read Latency (p99) | `http-read-latency-p99` | milliseconds | 99th percentile HTTP-level read latency. |
| HTTP Write Latency (max) | `http-write-latency-max` | milliseconds | Maximum HTTP-level write latency observed. |
| JVM Heap Pressure | `jvm-heap-pressure` | percentage (0-100) | Fleet-wide average JVM heap utilization (heap used / heap capacity) across container clusters. |
| Memory Utilization (max per cluster) | `memory-utilization-node-container` | percentage (0-100) | Maximum OS-level memory utilization across nodes in container clusters. |
| Memory Utilization (max per cluster) | `memory-utilization-node-content` | percentage (0-100) | Maximum OS-level memory utilization across nodes in content clusters. |
| QoS - Read (success %) | `qos-read-success` | percentage (0-100) | Percentage of read requests that did not return a 5xx server error. |
| QoS - Write (success %) | `qos-write-success` | percentage (0-100) | Percentage of write requests that did not return a 5xx server error. |
| Query Latency (avg) | `query-latency-avg` | milliseconds | Average query latency across the container cluster (total latency divided by request count). |
| Query Latency (max) | `query-latency-max` | milliseconds | Maximum query latency observed across the container cluster. |
| Restarts (1h) | `restarts` | count | Total number of Vespa service restarts across the fleet during the time window. |
###### Query
| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Degraded Queries (sum) | `degraded-queries-sum` | requests/second | Rate of queries that were degraded (soft-doomed or match-limited), summed across the container cluster. |
| Docs matched per query per rank profile | `docs-matched-per-query-per-rank-profile` | count | Average documents matched per query, aggregated across all rank profiles - a proxy for query selectivity. |
| Docsum executor accepted (rate) | `docsum-executor-accepted-rate` | operations/second | Average rate of docsum tasks accepted by the per-node docsum executor across content nodes. |
| Docsum executor queue size (max) | `docsum-executor-queue-size-max` | count | Peak length of the per-node docsum thread-pool queue on any single content node. |
| Docsum latency | `docsum-latency-avg` | seconds | Average summary-rendering (docsum) latency across content nodes, in seconds. |
| Docsum latency | `docsum-latency-max` | seconds | Peak summary-rendering (docsum) latency observed on any content node, in seconds. |
| Document summaries requested (rate) | `document-summaries-requested-rate` | requests/second | Document summaries requested per second across content nodes, derived from the docsum-latency sample count. |
| Documents Coverage | `documents-coverage` | percentage (0-100) | Percentage of the document corpus actually covered by search (documents covered vs. documents total). |
| Documents Matched (rate) | `documents-matched-rate` | operations/second | Documents matched per second across the content cluster, summed across all content nodes and rank profiles. |
| Empty Results (sum) | `empty-results-sum` | requests/second | Rate of queries returning zero results, summed across the container cluster. |
| Failed Queries (sum) | `failed-queries-sum` | requests/second | Rate of queries that failed, summed across the container cluster. |
| Grouping time per rank profile (avg) | `grouping-time-per-rank-profile-avg` | milliseconds | Average post-match time per query (result construction, grouping, packing), aggregated across all rank profiles and content nodes. |
| Grouping time per rank profile (peak) | `grouping-time-per-rank-profile-peak` | milliseconds | Peak post-match (grouping/result-construction) time observed in any snapshot interval, aggregated across all rank profiles. |
| Hits per Query (avg) | `hits-per-query-avg` | count | Average number of hits returned per query, weighted across the container cluster. |
| Match executor accepted (rate) | `match-executor-accepted-rate` | operations/second | Average rate of tasks accepted into the match executor across content nodes. |
| Match executor queue size (max) | `match-executor-queue-size-max` | count | Peak match thread-pool queue depth on any single content node. |
| Match Executor Utilization | `match-executor-utilization-avg` | percentage (0-100) | Average match-thread-pool utilization across content nodes, as a percentage. |
| Match Executor Utilization | `match-executor-utilization-max` | percentage (0-100) | Peak match-thread-pool utilization on any single content node, as a percentage. |
| Matching Queries (rate) | `matching-queries-rate` | requests/second | Per-node-average rate of queries being matched on content nodes. |
| Matching Query Latency (avg) | `matching-query-latency-avg` | milliseconds | Average matching-phase query latency across all content nodes and rank profiles. |
| Matching Query Setup Time (avg) | `matching-query-setup-time-avg` | milliseconds | Average per-query setup overhead (query decode, blueprint build, rank setup) on content nodes. |
| Queries per second by content group (min) | `queries-per-second-by-content-group-min` | requests/second | Lowest per-content-group query rate observed, deliberately using min to catch a failed node emitting stale/frozen metric values. |
| Queries per second per rank profile | `queries-per-second-per-rank-profile` | requests/second | Average per-(node, rank-profile) query rate observed at the content nodes. |
| Query Container Latency (avg) | `query-container-latency-avg` | milliseconds | Time spent inside the container processing the query, excluding backend dispatch. |
| Query Error Breakdown (sum) | `query-error-breakdown-backend-comm` | requests/second | Rate of queries that failed due to backend (content-node) communication errors. |
| Query Error Breakdown (sum) | `query-error-breakdown-empty-docsums` | requests/second | Rate of queries that failed because document summaries could not be retrieved. |
| Query Error Breakdown (sum) | `query-error-breakdown-invalid-param` | requests/second | Rate of queries that failed due to an invalid query parameter. |
| Query Error Breakdown (sum) | `query-error-breakdown-timeout` | requests/second | Rate of queries that failed due to a timeout error. |
| Query Error Breakdown (sum) | `query-error-breakdown-unhandled` | requests/second | Rate of queries that failed due to an unhandled exception in the container. |
| Query Latency (p95) | `query-latency-p95` | milliseconds | Average 95th percentile query latency. |
| Query Latency (p99) | `query-latency-p99` | milliseconds | Average 99th percentile query latency. |
| Query Rate (QPS) | `query-rate-qps-avg` | requests/second | Total application-wide query rate, summed across all container instances. |
| Query Rate (QPS) | `query-rate-qps-peak` | requests/second | Highest per-host peak queries-per-second observed across the container cluster. |
| Query Timeout (p99) | `query-timeout-p99` | milliseconds | 99th percentile of the configured per-query timeout values actually applied by clients. |
| Rank Profile Query Latency (max) | `rank-profile-query-latency-max` | milliseconds | Peak per-query matching latency on content nodes, taken across all rank profiles combined. |
| Rerank time per rank profile (avg) | `rerank-time-per-rank-profile-avg` | milliseconds | Average second-phase rerank time per query, aggregated across all rank profiles and content nodes (no non-rank-profile-labeled equivalent metric exists to prefer instead). |
| Rerank time per rank profile (peak) | `rerank-time-per-rank-profile-peak` | milliseconds | Peak second-phase rerank time observed in any snapshot interval, aggregated across all rank profiles. |
| search-handler — Utilization | `search-handler-utilization-avg` | percentage (0-100) | Average utilization of the container's search-handler thread pool. |
| search-handler — Utilization | `search-handler-utilization-max` | percentage (0-100) | Peak utilization of the container's search-handler thread pool on any single host. |
| Search Protocol Query Latency (avg) | `search-protocol-query-latency-avg` | milliseconds | Average query latency measured at the search-protocol level on content nodes. |
| Search Protocol Query Latency (max) | `search-protocol-query-latency-max` | milliseconds | Maximum query latency measured at the search-protocol level on any content node. |
| Soft doom factor per rank profile | `soft-doom-factor-per-rank-profile-avg` | count | Average adaptive soft-timeout multiplier per query, aggregated across all rank profiles. |
| Soft doom factor per rank profile | `soft-doom-factor-per-rank-profile-max` | count | Highest adaptive soft-timeout multiplier observed for any rank profile. |
| Soft doom factor per rank profile | `soft-doom-factor-per-rank-profile-min` | count | Lowest adaptive soft-timeout multiplier observed for any rank profile. |
| Soft-doomed queries per rank profile (sum) | `soft-doomed-queries-per-rank-profile-sum` | requests/second | Rate of queries that hit the soft timeout, summed across all rank profiles and content nodes. |
| Rank Profile Query Latency (avg) | `rank-profile-query-latency-avg` | milliseconds | Average per-query matching latency on content nodes, taken across all rank profiles combined. |
| Query setup time per rank profile (avg) | `query-setup-time-per-rank-profile-avg` | milliseconds | Average per-query setup overhead (query decode, blueprint build, rank setup, plus post-match finalisation) per rank profile. |
###### Feed
| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Container Feed Latency (max) | `container-feed-latency-max` | milliseconds | Maximum feed request latency observed at the container, across all operation types and clusters. |
| Container Feed Operations (sum) | `container-feed-operations-sum` | operations/second | Total feed operation rate (puts, updates, removes) handled by container clusters, summed across operation types. |
| Content: Commit Latency (avg) | `content-commit-latency-avg` | milliseconds | Average commit latency (time to commit fed changes to the index/store) on content nodes. |
| Content: Commit Operations (sum) | `content-commit-operations-sum` | operations/second | Rate of commit operations (batched writes applied to the index/store) on content nodes. |
| Content: Storage Feed Rates (sum) | `content-storage-feed-rates-put` | operations/second | Rate of PUT operations processed by the storage/persistence-engine layer on content nodes, counted per document replica. |
| Content: Storage Feed Rates (sum) | `content-storage-feed-rates-remove` | operations/second | Rate of REMOVE operations processed by the storage/persistence-engine layer on content nodes, counted per document replica. |
| Content: Storage Feed Rates (sum) | `content-storage-feed-rates-update` | operations/second | Rate of UPDATE operations processed by the storage/persistence-engine layer on content nodes, counted per document replica. |
| Content: Storage Put Latency (max) | `content-storage-put-latency-max` | milliseconds | Maximum PUT operation latency observed at the storage/persistence-engine layer on content nodes. |
| Content: Storage Update Latency | `content-storage-update-latency-avg` | milliseconds | Average UPDATE operation latency at the storage/persistence-engine layer on content nodes. |
| Content: Storage Update Latency | `content-storage-update-latency-max` | milliseconds | Maximum UPDATE operation latency observed at the storage/persistence-engine layer on content nodes. |
| Distributor Latency - gets (max) | `distributor-latency-gets-max` | milliseconds | Maximum get operation latency observed at the distributor level. |
| Distributor Latency - puts (max) | `distributor-latency-puts-max` | milliseconds | Maximum put operation latency observed at the distributor level. |
| Distributor Latency - removes (max) | `distributor-latency-removes-max` | milliseconds | Maximum remove operation latency observed at the distributor level. |
| Distributor Latency - updates (max) | `distributor-latency-updates-max` | milliseconds | Maximum update operation latency observed at the distributor level. |
| Distributor Latency - visitor (max) | `distributor-latency-visitor-max` | milliseconds | Maximum visitor operation latency observed at the distributor level. |
| Distributor Operation - gets - Failures (sum) | `distributor-operation-gets-failures-sum` | operations/second | Rate of failed get operations at the distributor level; non-zero values indicate issues with writes. |
| Distributor Operation - puts - Failures (sum) | `distributor-operation-puts-failures-sum` | operations/second | Rate of failed put operations at the distributor level; non-zero values indicate issues with writes. |
| Distributor Operation Rates - gets - OK (sum) | `distributor-operation-rates-gets-ok-sum` | operations/second | Rate of successfully completed (OK) get operations at the distributor level. |
| Distributor Operation Rates - puts - OK (sum) | `distributor-operation-rates-puts-ok-sum` | operations/second | Rate of successfully completed (OK) put operations at the distributor level. |
| Distributor Operation Rates - removes - OK (sum) | `distributor-operation-rates-removes-ok-sum` | operations/second | Rate of successfully completed (OK) remove operations at the distributor level. |
| Distributor Operation Rates - updates - OK (sum) | `distributor-operation-rates-updates-ok-sum` | operations/second | Rate of successfully completed (OK) update operations at the distributor level. |
| Distributor Operation Rates - visitor - OK (sum) | `distributor-operation-rates-visitor-ok-sum` | operations/second | Rate of successfully completed (OK) visitor operations at the distributor level. |
| Distributor Operation - removes - Failures (sum) | `distributor-operation-removes-failures-sum` | operations/second | Rate of failed remove operations at the distributor level; non-zero values indicate issues with writes. |
| Distributor Operation - updates - Failures (sum) | `distributor-operation-updates-failures-sum` | operations/second | Rate of failed update operations at the distributor level; non-zero values indicate issues with writes. |
| Distributor Operation - visitor - Failures (sum) | `distributor-operation-visitor-failures-sum` | operations/second | Rate of failed visitor operations at the distributor level; non-zero values indicate issues with writes. |
| Document Processing Latency (avg) | `document-processing-latency-avg` | milliseconds | Average document processing (docproc chain) latency. |
| Document Processing Rate (sum) | `document-processing-rate-sum` | operations/second | Rate of documents processed through document processing (docproc) chains, summed across all chains and clusters. |
| Feed Blocked (nodes above resource limit) | `feed-blocked-nodes-above-resource-limit` | count | Number of content nodes currently exceeding a resource usage limit; when greater than zero, feeding is blocked for that cluster. |
| Feed Latency (avg) | `feed-latency-avg-ms` | milliseconds | Average feed latency, in milliseconds, for successful feed operations. |
| HTTP API Feed Rates (sum) | `http-api-feed-rates-put` | operations/second | Rate of PUT operations received via the Document/HTTP API. |
| HTTP API Feed Rates (sum) | `http-api-feed-rates-remove` | operations/second | Rate of REMOVE operations received via the Document/HTTP API. |
| HTTP API Feed Rates (sum) | `http-api-feed-rates-update` | operations/second | Rate of UPDATE operations received via the Document/HTTP API. |
| HTTP API Latency (avg) | `http-api-latency-avg` | milliseconds | Average latency of Document/HTTP API requests, distinct from the generic jdisc HTTP layer latency. |
| HTTP API Pending Requests (max) | `http-api-pending-requests-max` | count | Maximum number of Document/HTTP API requests pending (in-flight, not yet completed) on the container. |
| HTTP API Success vs Failures (sum) | `http-api-success-vs-failures-failed` | operations/second | Rate of failed Document/HTTP API operations. |
| HTTP API Success vs Failures (sum) | `http-api-success-vs-failures-parse-error` | operations/second | Rate of Document/HTTP API operations rejected due to document parse errors. |
| HTTP API Success vs Failures (sum) | `http-api-success-vs-failures-success` | operations/second | Rate of successful Document/HTTP API operations. |
| HTTP Write Latency (avg) | `http-write-latency-avg` | milliseconds | Average end-to-end HTTP write request latency at the container's HTTP layer. |
| HTTP Write Latency (p95) | `http-write-latency-p95` | milliseconds | 95th percentile HTTP write request latency at the container's HTTP layer. |
| HTTP Write Latency (p99) | `http-write-latency-p99` | milliseconds | 99th percentile HTTP write request latency at the container's HTTP layer. |
| Memory Index — Documents (per document type) | `memory-index-docs-in-memory-max` | count | Maximum number of documents held in the in-memory index on content nodes, summed per content group (to combine per-node/stripe values) and maxed across groups (since each group holds a full replica) to avoid double-counting. |
| Persistence engine input queue | `persistence-engine-input-queue-avg` | count | Average depth of the per-node persistence engine's input operation queue (operations waiting to be dispatched to Proton) over the time window. |
| Persistence engine input queue | `persistence-engine-input-queue-max` | count | Maximum depth of the per-node persistence engine's input operation queue observed during the time window. |
| Persistence engine throttle saturation (max) | `persistence-engine-throttle-saturation-active` | count | Maximum number of concurrently active (in-flight) operations in the content node's persistence engine. |
| Persistence engine throttle saturation (max) | `persistence-engine-throttle-saturation-throttle` | count | Maximum size of the adaptive throttle window that caps concurrent operations in the content node's persistence engine. |
###### Nearest Neighbor Search
| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| ANN Timeout Rate (sum) | `ann-timeout-rate-sum` | requests/second | Rate of queries per second that terminated an approximate NNS search early after hitting the ANN timeout. |
| Approximate NNS Distances Computed (rate) | `approximate-nns-distances-computed-rate` | operations/second | Rate of distance computations performed during approximate NNS across the content cluster. |
| Approximate NNS Nodes Visited (rate) | `approximate-nns-nodes-visited-rate` | operations/second | Rate of HNSW graph nodes visited during approximate NNS across the content cluster. |
| Query Rate of Approximate Nearest Neighbor Search Queries | `approximate-nns-query-rate` | requests/second | Rate of incoming queries per second that perform an approximate nearest-neighbor search. |
| Query Approximate Nearest Neighbor Search Time | `approximate-nns-time-avg` | milliseconds | Average time spent performing approximate NNS, measured only across queries that actually perform it. |
| Query Approximate Nearest Neighbor Search Time | `approximate-nns-time-max` | milliseconds | Maximum time observed for a single query's approximate NNS work, on any content node. |
| Approximate NNS Visit Efficiency (nodes visited / distances computed) | `approximate-nns-visit-efficiency` | ratio (dimensionless) | Ratio of HNSW graph nodes visited to distances computed during approximate NNS; values above 1.0 indicate the filter-first (Acorn-1) heuristic is active. |
| Buckets Pending Merge (sum) | `buckets-pending-merge-sum` | count | Number of buckets currently pending merge across distributors, which rises during data redistribution or node recovery. |
| Documents Active (per document type) | `documents-active-total` | count | Total number of active documents (primary copies) across all document types and content nodes in the cluster. |
| Documents Ready (per document type) | `documents-ready-total` | count | Total number of ready (searchable) documents across all document types and content nodes in the cluster. |
| Exact NNS Distance Computation Ratio (exact / total) | `exact-nns-distance-ratio-pct` | percentage (0-100) | Share of NNS distance computations done via exact (brute-force) search rather than approximate search; high values may indicate the approximate-threshold is set too high. |
| Exact NNS Distances Computed (rate) | `exact-nns-distances-computed-rate` | operations/second | Rate of distance computations performed during exact (brute-force) NNS across the content cluster. |
| Query Latency | `nns-query-latency-avg` | milliseconds | Average total time (matching plus ranking) spent per query on the content nodes, specific to the NNS/matching pipeline. |
| Query Latency | `nns-query-latency-max` | milliseconds | Maximum total time (matching plus ranking) observed for a single query on any content node, specific to the NNS/matching pipeline. |
| Query Rate of Non-Approximate Nearest Neighbor Search Queries | `non-approximate-nns-query-rate` | requests/second | Rate of incoming queries per second that do not perform approximate NNS, including queries that fall back to exact search. |
| Query Matching Time | `query-matching-time-avg` | milliseconds | Average time spent on the core query matching process, computed as query latency minus query setup time; includes exact NNS. |
| Query Matching Time | `query-matching-time-max` | milliseconds | Approximate maximum time spent on the core query matching process, computed as max query latency minus max query setup time (not a directly measured value, since the two maxima may occur on different queries). |
| Query Rate | `query-rate` | requests/second | Rate of incoming queries per second against the content cluster's document matching pipeline (equal to approximate-nns-query-rate plus non-approximate-nns-query-rate). |
| Query Setup Time | `query-setup-time-avg` | milliseconds | Average time spent setting up and tearing down queries, including filter computation and approximate NNS overhead. |
| Query Setup Time Without Approximate Nearest Neighbor Search Time | `query-setup-time-excl-ann-avg` | milliseconds | Average time spent on query setup and teardown excluding approximate NNS time, isolating filter computation and other overhead. |
| Query Setup Time | `query-setup-time-max` | milliseconds | Maximum observed time spent setting up and tearing down a single query on any content node. |
| Soft Timeout Rate (sum) | `soft-timeout-rate-sum` | requests/second | Rate of queries per second terminated early due to hitting the soft timeout, aggregated across rank profiles. |
| Thread Pool - Match Utilization | `thread-pool-match-utilization-avg` | percentage (0-100) | Average utilization of the match thread pool across content nodes. |
| Thread Pool - Match Utilization | `thread-pool-match-utilization-max` | percentage (0-100) | Peak utilization of the match thread pool observed on any content node. |
| Total Distances Computed (sum) | `total-distances-computed-sum` | operations/second | Total NNS distance computations per second, combining exact and approximate search, as an overall indicator of NNS-related CPU cost. |
###### Content Node
| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Attribute Resource Usage - Feeding Blocked (max) | `attribute-feeding-blocked-pct` | percentage (0-100) | Percentage of the time window during which feeding was blocked due to attribute resource usage limits on any content node. |
| Bucket Move Pending (max) | `bucket-move-pending-max` | count | Maximum number of buckets pending a move operation (e.g. during cluster resizing or redistribution) on any content node. |
| Document Store Cache Hit Rate (max) | `document-store-cache-hit-rate-max` | percentage (0-100) | Cache hit rate for the ready document store's summary-field cache, as a percentage; low values on quiescent clusters often show up as elevated CPU IOWait from disk-backed docsum reads. |
| Document Store Disk Usage (max) | `document-store-disk-usage-max` | bytes | Maximum disk space used by the ready document store on any content node. |
| Documents Active (per document type) | `documents-count-active` | count | Number of active/searchable document copies, summed across content clusters after taking the max across replica groups within each cluster to avoid double counting. |
| Documents Ready (per document type) | `documents-count-ready` | count | Number of documents available for search (ready), summed across content clusters after taking the max across replica groups within each cluster to avoid double counting. |
| Field Writer Utilization & Saturation | `field-writer-saturation-max` | percentage (0-100) | Maximum observed saturation of Proton's field writer executor thread pool; high saturation indicates a write bottleneck. |
| Field Writer Utilization & Saturation | `field-writer-utilization-avg` | percentage (0-100) | Average utilization of Proton's field writer executor thread pool across content nodes. |
| Filestor Average Queue Wait (avg) | `filestor-average-queue-wait-avg` | milliseconds | Average time operations spent waiting in the filestor queue across content nodes. |
| Filestor Queue Size (max) | `filestor-queue-size-max` | count | Maximum filestor queue size observed on any content node; a growing queue indicates storage cannot keep up with incoming operations. |
| Maintenance Job Activity (max) | `maintenance-job-activity-attr-flush` | percentage (0-100) | Activity level of the attribute-flush maintenance job (0% = idle, 100% = fully active), maximum across content nodes. |
| Maintenance Job Activity (max) | `maintenance-job-activity-disk-idx-fusion` | percentage (0-100) | Activity level of the disk-index-fusion maintenance job (0% = idle, 100% = fully active), maximum across content nodes. |
| Maintenance Job Activity (max) | `maintenance-job-activity-docstore-compact` | percentage (0-100) | Activity level of the document-store-compaction maintenance job (0% = idle, 100% = fully active), maximum across content nodes. |
| Maintenance Job Activity (max) | `maintenance-job-activity-lid-compact` | percentage (0-100) | Activity level of the LID (local document id) space-compaction maintenance job (0% = idle, 100% = fully active), maximum across content nodes. |
| Maintenance Job Activity (max) | `maintenance-job-activity-mem-idx-flush` | percentage (0-100) | Activity level of the memory-index-flush maintenance job (0% = idle, 100% = fully active), maximum across content nodes. |
| Proton Disk Usage (avg) | `proton-disk-usage-avg` | percentage (0-100) | Relative disk usage of the content node's Proton process, as a percentage of the available disk; 100% means feeding is blocked. |
| Proton Executor Utilization | `proton-executor-utilization-avg` | percentage (0-100) | Average utilization of Proton's own executor thread pool across content nodes. |
| Proton Executor Utilization | `proton-executor-utilization-max` | percentage (0-100) | Maximum observed utilization of Proton's own executor thread pool on any content node. |
| Proton Memory Usage (avg) | `proton-memory-usage-avg` | percentage (0-100) | Relative memory usage of the content node's Proton process, as a percentage of the available memory. |
| Shared Executor Queue Size (max) | `shared-executor-queue-size-max` | count | Maximum shared executor queue size observed on any content node during the time window. |
| Shared Executor Utilization | `shared-executor-utilization-avg` | percentage (0-100) | Average utilization of Proton's shared executor thread pool across content nodes. |
| Shared Executor Utilization | `shared-executor-utilization-max` | percentage (0-100) | Maximum observed utilization of Proton's shared executor thread pool on any content node. |
###### Resources
| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| CPU IOWait (avg/max/min) | `cpu-iowait-avg` | percentage (0-100) | Average per-host CPU I/O-wait percentage across the cluster over the time window, indicating time spent waiting on disk I/O rather than doing CPU work. |
| CPU IOWait (avg/max/min) | `cpu-iowait-max` | percentage (0-100) | Maximum per-host CPU I/O-wait percentage across the cluster over the time window; high values point to disk being the bottleneck rather than CPU. |
| CPU IOWait (avg/max/min) | `cpu-iowait-min` | percentage (0-100) | Minimum per-host CPU I/O-wait percentage across the cluster over the time window. |
| CPU Throttled Time (sum) | `cpu-throttled-time-sum` | count | Sum of per-node CPU cgroup-throttling rate across the cluster over the time window; non-zero values indicate nodes are hitting their CPU resource limits. |
| CPU Utilization (max per cluster) | `cpu-utilization-container` | percentage (0-100) | Maximum CPU utilization across the nodes of the container cluster. |
| CPU Utilization (max per cluster) | `cpu-utilization-content` | percentage (0-100) | Maximum CPU utilization across the nodes of the content cluster. |
| default-handler-common — Utilization | `default-handler-common-utilization-avg` | percentage (0-100) | Average active threads as a percentage of pool size for the default-handler-common thread pool over the time window. |
| default-handler-common — Utilization | `default-handler-common-utilization-max` | percentage (0-100) | Peak active threads as a percentage of pool size for the default-handler-common thread pool over the time window. |
| default-handler-common — Work Queue Size | `default-handler-common-work-queue-size-avg` | count | Average number of pending tasks queued for the default-handler-common thread pool over the time window. |
| default-handler-common — Work Queue Size | `default-handler-common-work-queue-size-max` | count | Peak number of pending tasks queued for the default-handler-common thread pool observed during the time window. |
| default-handler-common — Work Queue Utilization | `default-handler-common-work-queue-utilization-avg` | percentage (0-100) | Average queue fill percentage (size / capacity) for the default-handler-common thread pool over the time window. |
| default-handler-common — Work Queue Utilization | `default-handler-common-work-queue-utilization-max` | percentage (0-100) | Peak queue fill percentage (size / capacity) for the default-handler-common thread pool observed during the time window. |
| Disk Utilization (avg/max/min) | `disk-utilization-avg` | percentage (0-100) | Average OS-level disk utilization percentage across the cluster's nodes. |
| Disk Utilization (avg/max/min) | `disk-utilization-max` | percentage (0-100) | Maximum OS-level disk utilization percentage across the cluster's nodes; sustained high values risk feed/query slowdowns from disk saturation. |
| Disk Utilization (avg/max/min) | `disk-utilization-min` | percentage (0-100) | Minimum OS-level disk utilization percentage across the cluster's nodes. |
| feedapi-handler — Utilization | `feedapi-handler-utilization-avg` | percentage (0-100) | Average active threads as a percentage of pool size for the feedapi-handler thread pool over the time window. |
| feedapi-handler — Utilization | `feedapi-handler-utilization-max` | percentage (0-100) | Peak active threads as a percentage of pool size for the feedapi-handler thread pool over the time window. |
| feedapi-handler — Work Queue Size | `feedapi-handler-work-queue-size-avg` | count | Average number of pending tasks queued for the feedapi-handler thread pool over the time window. |
| feedapi-handler — Work Queue Size | `feedapi-handler-work-queue-size-max` | count | Peak number of pending tasks queued for the feedapi-handler thread pool observed during the time window. |
| feedapi-handler — Work Queue Utilization | `feedapi-handler-work-queue-utilization-avg` | percentage (0-100) | Average queue fill percentage (size / capacity) for the feedapi-handler thread pool over the time window. |
| feedapi-handler — Work Queue Utilization | `feedapi-handler-work-queue-utilization-max` | percentage (0-100) | Peak queue fill percentage (size / capacity) for the feedapi-handler thread pool observed during the time window. |
| GPU Memory Utilization (max) | `gpu-memory-utilization-max` | percentage (0-100) | Maximum GPU memory utilization (used/total) across GPU-equipped nodes, averaged over the time window. |
| GPU Utilization (max) | `gpu-utilization-container-max` | percentage (0-100) | Maximum GPU utilization across container-cluster nodes with GPUs attached, averaged over the time window (only produces data for GPU-equipped nodes). |
| JVM Direct Memory (avg) | `jvm-direct-memory-capacity` | bytes | Average configured JVM direct (off-heap) memory capacity on container nodes. |
| JVM Direct Memory (avg) | `jvm-direct-memory-used` | bytes | Average JVM direct (off-heap) memory currently in use on container nodes. |
| JVM GC Overhead (max) | `jvm-gc-overhead-max` | percentage (0-100) | Approximate percentage of CPU time spent in JVM garbage collection on container nodes, averaged over the time window. |
| JVM GC Pause Duration | `jvm-gc-pause-duration-avg` | seconds | Average stop-the-world JVM GC pause duration on container nodes over the time window. |
| JVM GC Pause Duration | `jvm-gc-pause-duration-max` | seconds | Worst single stop-the-world JVM GC pause observed on container nodes during the time window. |
| JVM Heap Usage (avg) | `jvm-heap-usage-capacity` | bytes | Average configured JVM heap memory capacity (max heap size) on container nodes. |
| JVM Heap Usage (avg) | `jvm-heap-usage-used` | bytes | Average JVM heap memory currently in use on container nodes. |
| JVM Native Memory (avg) | `jvm-native-memory-avg` | bytes | Average native (off-JVM) memory used by C++ libraries loaded via JNI (e.g. ONNX Runtime for embedders, linguistics) on container nodes. |
| Memory Utilization (avg/max/min) | `memory-node-utilization-avg` | percentage (0-100) | Average OS-level memory utilization across the cluster's nodes (physical RAM usage), distinct from the Vespa-computed content memory-utilization metric. |
| Memory Utilization (avg/max/min) | `memory-node-utilization-max` | percentage (0-100) | Maximum OS-level memory utilization across the cluster's nodes (physical RAM usage), distinct from the Vespa-computed content memory-utilization metric. |
| Memory Utilization (avg/max/min) | `memory-node-utilization-min` | percentage (0-100) | Minimum OS-level memory utilization across the cluster's nodes (physical RAM usage). |
| Resource Usage: Max Memory Utilization vs Limit | `memory-utilization-content` | percentage (0-100) | Maximum memory utilization of the content cluster's cluster controller. |
| Network Throughput (bytes received/sent) | `network-throughput-bytes-received-sent-received` | bytes | Average per-interval network bytes received by container nodes, summed across the cluster over the time window. |
| Network Throughput (bytes received/sent) | `network-throughput-bytes-received-sent-sent` | bytes | Average per-interval network bytes sent by container nodes, summed across the cluster over the time window. |
| Open Server Connections (max) | `open-server-connections-max` | count | Peak number of open HTTP server connections on container nodes observed during the time window. |
| Requests per HTTP Connection | `requests-per-http-connection-avg` | count | Average number of HTTP requests served per connection over its lifetime on container nodes during the time window. |
| Requests per HTTP Connection | `requests-per-http-connection-max` | count | Peak number of HTTP requests served by a single connection over its lifetime on container nodes during the time window. |
| search-handler — Work Queue Size | `search-handler-work-queue-size-avg` | count | Average number of pending tasks queued for the search-handler thread pool over the time window. |
| search-handler — Work Queue Size | `search-handler-work-queue-size-max` | count | Peak number of pending tasks queued for the search-handler thread pool observed during the time window. |
| search-handler — Work Queue Utilization | `search-handler-work-queue-utilization-avg` | percentage (0-100) | Average queue fill percentage (size / capacity) for the search-handler thread pool over the time window. |
| search-handler — Work Queue Utilization | `search-handler-work-queue-utilization-max` | percentage (0-100) | Peak queue fill percentage (size / capacity) for the search-handler thread pool observed during the time window. |
###### Health
| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Cluster State Changes (sum) | `cluster-state-changes-sum` | count | Number of cluster state changes observed in the given time window. Frequent changes are most often caused by Vespa upgrades or required node restarts. |
| Core Dumps (1h) | `core-dumps-processed-max` | count | Number of core dumps processed across all nodes in the given time window; should be 0. |
| Deactivated Containers | `deactivated-containers` | count | Average number of deactivated container instances per sample, over the given time window. Brief spikes during deployments are normal; persistently high values may indicate resource leaks. |
| Estimated Time to In-Sync | `estimated-time-to-in-sync` | seconds | Rough estimate, from the recent (fixed 30-minute) convergence rate, of the time remaining until all buckets are back in sync; capped at 7 days. |
| Failed Component Graphs | `failed-component-graphs` | operations/second | Rate of failed application component-graph reconfigurations, averaged over the given time window; nonzero after a deployment may indicate a configuration or component issue. |
| HTTP Requests Prematurely Closed | `http-requests-prematurely-closed` | operations/second | Rate of HTTP requests closed by the client before the server finished responding, averaged over the given time window. |
| Merge Bucket Pending | `merge-bucket-pending` | count | Number of pending bucket merge operations across all distributors, summed per application. |
| Node count (max) | `node-count-max` | count | Total number of nodes currently in the active state for the application. |
| Node Events (sum) | `node-events-sum` | count | Number of node events reported by the cluster controller in the given time window. |
| Node state: down (max) | `node-state-down-max` | count | Number of content nodes in the 'down' state per cluster (nodes considered unavailable by the cluster controller). |
| Node state: initializing (max) | `node-state-initializing-max` | count | Number of content nodes in the 'initializing' state per cluster (nodes currently starting up and not yet serving). |
| Node state: maintenance (max) | `node-state-maintenance-max` | count | Number of content nodes in the 'maintenance' state per cluster (nodes intentionally taken down for maintenance). |
| Node state: retired (max) | `node-state-retired-max` | count | Number of content nodes in the 'retired' state per cluster (nodes being decommissioned). |
| Node state: up (max) | `node-state-up-max` | count | Number of content nodes in the 'up' state per cluster. |
| Re-indexing remaining | `re-indexing-remaining` | percentage (0-100) | Worst-case document reindexing progress remaining across document types, as a percentage (100 = not started, 0 = complete or no reindex running). |
| Resource Usage: Attribute Address Space | `resource-usage-attribute-address-space` | percentage (0-100) | Worst-case attribute address-space utilization across content nodes; feed is blocked when this approaches 100%. |
| Resource Usage: Max Disk Utilization vs Limit | `resource-usage-max-disk-utilization-vs-limit` | percentage (0-100) | Maximum disk utilization vs the configured limit; feed is blocked when utilization exceeds the limit. |
| Service Restarts (total) | `service-restarts-total` | count | Total service restarts across all nodes, as tracked by the config sentinel. Increases indicate services crashing or being restarted. |
| Rate-Limited Requests | `rate-limited-requests-sum` | operations/second | Rate of requests checked by the RateLimitingSearcher (i.e. requests with rate.id and rate.quota set); only present if the searcher is in use. |
| Requests Rejected (Over Quota) | `requests-rejected-over-quota-sum` | operations/second | Rate of requests rejected by the RateLimitingSearcher for exceeding their quota (rate.quota); in dry-run mode these are counted but not actually rejected. |
###### Autoscaling: Content
| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Active nodes (max) | `active-nodes-count` | count | Maximum number of active nodes in the content cluster during the test window. |
| Active nodes (max) | `active-nodes-non-active-fraction` | fraction (0-1) | Fraction of content-cluster nodes that are not yet active, indicating an in-progress resize or data redistribution. |
| Autoscaling actions (sum) | `autoscaling-actions-sum` | count | Number of autoscaler rescale actions taken for the content cluster during the test window. |
| CPU load adjustment (peak / ideal) | `cpu-load-adjustment-ratio` | ratio (peak / ideal, 1.0 = ideally scaled) | Ratio of peak to ideal CPU load for the content cluster; above 1.0 signals scale-up pressure and below 1.0 signals the cluster is over-provisioned for CPU. |
| CPU load — peak vs ideal (max) | `cpu-load-peak-vs-ideal-ideal` | percentage (0-100) | The autoscaler's ideal CPU load target for the content cluster, as a percentage (already includes growth and failover headroom, so normally below 100). |
| CPU load — peak vs ideal (max) | `cpu-load-peak-vs-ideal-peak` | percentage (0-100) | The autoscaler's peak observed CPU load for the content cluster, as a percentage, from the config server's cluster.load.peak.cpu signal. |
| Query growth (rate vs 3h average) | `query-growth-rate-vs-3h-average-content` | ratio (test-window average / trailing 3h average) | Ratio of the content cluster's query rate during the test window to its own trailing 3-hour average, a proxy for the autoscaler's internal query-growth headroom factor. |
| Query vs write rate — CPU mix driver (sum) | `query-vs-write-rate-cpu-mix-driver-query-content` | operations/second | Query rate for the content cluster, one half of the query/write mix that determines the autoscaler's ideal CPU target. |
| Query vs write rate — CPU mix driver (sum) | `query-vs-write-rate-cpu-mix-driver-write-content` | operations/second | Write rate (put + update + remove operations per second) for the content cluster, the other half of the query/write mix that determines the autoscaler's ideal CPU target. |
###### Autoscaling: Container
| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Active nodes (max) | `active-nodes-container` | count | Maximum number of active container nodes in the cluster during the test window. |
| Active nodes (max) | `non-active-node-fraction-container` | fraction | Peak fraction of non-active (mid-transition) nodes in the container cluster during the test window, non-zero while a resize is in progress. |
| Query growth (rate vs 3h average) | `query-growth-rate-vs-3h-average` | ratio | Ratio of the container cluster's query rate during the test window to its trailing 3-hour average, a proxy for the autoscaler's growth-headroom signal (values above 1.0 indicate rising traffic). |
| Query vs write rate — CPU mix driver (sum) | `query-vs-write-rate-cpu-mix-driver-query` | operations/second | Average query rate across the container cluster during the test window; one side of the query/write mix that shapes the autoscaler's ideal CPU target. |
| Query vs write rate — CPU mix driver (sum) | `query-vs-write-rate-cpu-mix-driver-write` | operations/second | Average write (feed) operation rate across the container cluster during the test window; the other side of the query/write mix that shapes the autoscaler's ideal CPU target. |
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Production tests](#page-title)
- [Production test setup](#normal-flow)
- [Test suites](#test-suites)
- [Test file structure](#test-file-structure)
- [Test specification](#test-yaml-and-json-specification)
- [Examples](#examples)
- [Metric presets](#metric-presets)
- [Overview](#metric-presets-overview)
- [Query](#metric-presets-query)
- [Feed](#metric-presets-feed)
- [Nearest Neighbor Search](#metric-presets-nearest-neighbor-search)
- [Content Node](#metric-presets-content-node)
- [Resources](#metric-presets-resources)
- [Health](#metric-presets-health)
- [Autoscaling: Content](#metric-presets-autoscaling-content)
- [Autoscaling: Container](#metric-presets-autoscaling-container)
---
### Basic HTTP testing
[](/en/reference/applications/testing.html.md "View as Markdown")
#### Basic HTTP testing
[](/en/reference/applications/testing.html.md "View as Markdown")
This is the Vespa Testing reference for basic HTTP tests, used to write [Vespa application system tests](../../applications/testing.html).
These tests verify the behaviour of a Vespa application by using its HTTP interfaces. Basic HTTP tests are written in JSON; to write more advanced tests, see the [Java testing reference](testing-java.html).
See the [testing guide](../../applications/testing.html) for examples of how to run the tests.
##### Test suites
The [testing documentation](../../applications/testing.html) defines three test scenarios, comprised of three test code categories. For basic HTTP tests, the category of a test is defined by its placement in the application tests directory:
| Category | Directory | Description |
| --- | --- | --- |
| System test | tests/system-test/ | Independent, functional tests |
| Staging setup | tests/staging-setup/ | Set state before upgrade |
| Staging test | tests/staging-test/ | Verify state after upgrade |
**Note:** If the application package has Java code, the `tests` directory is `src/test/application/tests`
See the [production test reference](testing-production.html) for the `tests/production-test/` category, which uses a different, YAML or JSON based, format.
Each test is described by a JSON file, and may include other files using relative paths:
```
$ ls -1 tests/*/*
tests/staging-setup/set-up-old-documents.json
tests/staging-test/verify-search-still-works.json
tests/system-test/data/document.json
tests/system-test/feed-and-search-test.json
tests/system-test/ranking-test.json
```
For an example with actual system and staging tests, check out a Vespa Cloud [sample test suite](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/CI-CD).
##### Test file structure
Each `.json` file directly under any of the directories listed above describes one test. Each test consists of a series of steps, and each step specifies an HTTP request to run and some assertions about the response to obtain. Some additional properties may also be specified on both the test and step levels. A full example, with `//` comments:
```
{
"name": "my test",
"defaults": {
"cluster": "default",
"parameters": {
"timeout": "1.618s"
}
},
"steps": [
{
"name": "clear existing documents",
"request": {
"method": "DELETE",
"uri": "/document/v1/",
"parameters": {
"cluster": "music",
"selection": "true"
}
}
},
{
"name": "feed foo",
"request": {
"method": "POST",
// should contain payload as expected by /document/v1/
"body": "foo/body.json",
// specify only the path and query for Vespa requests
"uri": "/document/v1/test/music/docid/foo?timeout=8s",
// JSON object file; merged with query from "uri"
"parameters": "foo/parameters.json"
}
// no response spec: just assert code 200
},
{
"name": "query for foo",
"request": {
// no "uri": defaults to "/search/"
"parameters": {
"query": "artist: foo"
}
},
"response": {
"body": {
"root": {
"children": [
// assert "children" has a single element ...
{
// ... which has the field "fields" ...
"fields": {
// ... where the field "artist" is "Foo Fighters" ...
"artist": "Foo Fighters"
},
// ... and the field "relevance" close to 0.381862383599
"relevance": 0.381862383599
}
]
}
}
}
}
]
}
```
###### Test JSON specification
A full list of fields, with description:
| Name | Parent | Type | Default | Description |
| --- | --- | --- | --- | --- |
| name | root
step | string | file name, step _n_ | Name used for display purposes in the test report. The file name is used by default for the test, while the 1-indexed "step n" is used for steps. |
| defaults | root | object | | Default settings for all steps in this test. May be overridden in each step. |
| steps | root | array | | The non-empty list of steps that constitute this test. |
| request | step | object | | A specification of a request to send, to Vespa, or to an external service. |
| cluster | defaults request | string | | The name of the Vespa cluster to send a request to, as specified in [services.xml](services/services.html). If this is not specified, and the application has a single container cluster, this is used. |
| method | request | string | "GET" | The HTTP method to use for a request. |
| uri | request | string | "/search/" | When this is path + (encoded) query, the host is determined by the specified cluster; otherwise, it must be an absolute URI (with scheme), and its host is used. Query parameters specified here override those specified in the defaults. |
| parameters | defaults request | string object | | HTTP request query parameters. The values should not be encoded. These are merged with parameters from the specified URI, and override those specified in the defaults. If the value is a string, it must be a relative file reference to a parameters object. |
| body | request response | string object | | The body for a request, or the partial body (see [matching](#json-matching)) for a response. If the value is a string, it must be a relative file reference to a JSON object to be used in its place. |
| response | step | object | | A specification for assertions to make on the body of the HTTP response obtained by executing the HTTP request in the same step. |
| code | response | number | 200 | The status code the response should have. |
###### JSON matching
All requests and responses must be in JSON format. The tests allow simple JSON verification, by describing _what should be present_ in the actual response. This is done by specifying a JSON structure, a _template_, for each response, and requiring each field present in the template to match fields in the actual response. Unmatched fields result in a test failure, with the following rules:
- Objects must contain all listed fields, and may also contain unlisted ones.
- Arrays must match element-by-element.
- Numbers must match within precision `1e-9`.
- All other values must match exactly.
Note that the empty object `{ }` matches any other object, and can be used to fill elements of an array that require no further validation.
Copyright © 2026 - [Cookie Preferences](#)
---
### validation-overrides.xml
[](/en/reference/applications/validation-overrides.html.md "View as Markdown")
#### validation-overrides.xml
[](/en/reference/applications/validation-overrides.html.md "View as Markdown")
_validation-overrides.xml_ is added to the root of an [application package](application-packages.html) (i.e. next to [services.xml](services/services.html)) to allow a deployment that otherwise fails to validate to proceed. The message will explain what the validation concerns and how to add a validation override
Validations protect against inadvertently corrupting a production instance. Overriding them may be useful e.g. if the application is not in production yet or if you think the consequences of inconsistencies or loss of the data in a particular field are fine.
Read more about schema changes in the [schema reference](../schemas/schemas.html#modifying-schemas).
##### Structure
```xml
validation-id
```
Any number of `allow` tags is permissible. Example:
```xml
resources-reduction
field-type-change
```
##### allow
An `allow` tag disables a particular validation for a limited time and contains a single validation id. `allow` tags with unknown ids are ignored.
| Attribute | Mandatory | Value |
| --- | --- | --- |
| until | Yes | The last day this change is allowed, as a ISO-8601-format date in UTC, e.g. 2016-01-30. Dates may at most be 30 days in the future, but should be as close to now as possible for safety, while allowing time for review and propagation to all deployed zones. `allow`-tags with dates in the past are ignored. |
| comment | No | Text explaining the reason for the change to humans. |
##### List of validation overrides
See [ValidationId.java](https://github.com/vespa-engine/vespa/blob/master/config-model-api/src/main/java/com/yahoo/config/application/api/ValidationId.java) for a complete list of validation overrides.
Copyright © 2026 - [Cookie Preferences](#)
---
## Agent Ready Schemas
### Agent-ready schemas
[](/en/rag/agent-ready-schemas.html.md "View as Markdown")
#### Agent-ready schemas
[](/en/rag/agent-ready-schemas.html.md "View as Markdown")
Agents will increasingly be the users of search applications. They differ from the typical human user in two ways:
- they need all the information on how to use a particular application in their context
- they can use that information to generate complex, high-precision queries to solve tasks
Vespa provides the information agents need as a skill available on container endpoints. To enable it, add the `schemas` skill to `services.xml` in your search container cluster:
```xml
```
With this in place, you can point agents to `[endpoint]/skills/v1/schemas` so they can build precise queries against your search application. The skill provides information on which schemas, fields, rank profiles, and inputs are available, their types, match modes, and which embedder to use for fields whose embeddings are created by Vespa during indexing.
##### Schema explanations
In addition to the default information in the skill, you can provide _explanations_ to give agents information that can't be derived from the schema itself:
```vespa-schema-language
schema article {
explanation {
Scientific articles imported from pdf archives.
Individual fields denote whether the article is peer
reviewed, when it was published, by whom and so on. Titles,
abstracts and full text are searchable
both lexically and as multi-vector representations,
see fields for details.
}
document {
field publish_date type long {
indexing: attribute
explanation: Publish date as an epoch seconds timestamp.
}
…
}
rank-profile topical {
explanation {
Orders articles by broad relevance to the described concept,
emphasizing semantic matching while ranking exact lexical
matches higher.
@input query(embedding) the concept as a single embedding
@input query(freshness) the weight (0-1) to give to recency
among topical matches
}
…
}
}
```
Explanations can be added on schemas, fields, and rank profiles, and on rank profiles they can also contain explanations for the inputs to the profile. See the [schema reference](../reference/schemas/schemas.html).
Explanations should be used when there is _additional relevant information to provide_, such as the range and unit of values of a field, what are suitable use cases for a rank profile, and similar. They should not explain what's obvious, or repeat information that is provided automatically, such as the type.
Copyright © 2026 - [Cookie Preferences](#)
---
## Api
### Vespa API and interfaces
[](/en/reference/api/api.html.md "View as Markdown")
#### Vespa API and interfaces
[](/en/reference/api/api.html.md "View as Markdown")
##### Deployment and configuration
- [Deploy API](deploy-v2.html): Deploy [application packages](../../basics/applications.html) to configure a Vespa application
- [Config API](config-v2.html): Get and Set configuration
- [Tenant API](application-v2.html): Configure multiple tenants in the config servers
##### Document API
- [Reads and writes](../../writing/reads-and-writes.html): APIs and binaries to read and update documents
- [/document/v1/](document-v1.html): REST API for operations based on document ID (get, put, remove, update)
- [Feeding API](../../clients/vespa-feed-client.html): High performance feeding API, the recommended API for feeding data
- [JSON feed format](../schemas/document-json-format.html): The Vespa Document format
- [Vespa Java Document API](../../writing/document-api-guide.html)
##### Query and grouping
- [Query API](../../querying/query-api.html), [Query API reference](query.html)
- [Query Language](../../querying/query-language.html), [Query Language reference](../querying/yql.html), [Simple Query Language reference](../querying/simple-query-language.html), [Predicate fields](../../schemas/predicate-fields.html)
- [Vespa Query Profiles](../../querying/query-profiles.html)
- [Grouping API](../../querying/grouping.html), [Grouping API reference](../querying/grouping-language.html)
##### Processing
- [Vespa Processing](../../applications/processing.html): Request-Response processing
- [Vespa Document Processing](../../applications/document-processors.html): Feed processing
##### Request processing
- [Searcher API](../../applications/searchers.html)
- [Federation API](../../querying/federation.html)
- [Web service API](../../applications/web-services.html)
##### Result processing
- [Custom renderer API](../../applications/result-renderers.html)
##### Status and state
- [Health and Metric APIs](../../operations/metrics.html)
- [/cluster/v2 API](cluster-v2.html)
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Vespa API and interfaces](#page-title)
- [Deployment and configuration](#deployment-and-configuration)
- [Document API](#document-api)
- [Query and grouping](#query-and-grouping)
- [Processing](#processing)
- [Request processing](#request-processing)
- [Result processing](#result-processing)
- [Status and state](#status)
---
### /application/v2/tenant API reference
[](/en/reference/api/application-v2.html.md "View as Markdown")
#### /application/v2/tenant API reference
[](/en/reference/api/application-v2.html.md "View as Markdown")
This is the /application/v2/tenant API reference with examples for the HTTP REST API to [list](#list-tenants), [create](#create-tenant) and [delete](#delete-tenant) a tenant, which can be used to [deploy](deploy-v2.html) an application.
The response format is JSON. The tenant value is "default".
The current API version is 2. The API port is 19071 - use [vespa-model-inspect](/en/reference/operations/self-managed/tools.html#vespa-model-inspect) service configserver to find config server hosts - example: `http://myconfigserver.mydomain.com:19071/application/v2/tenant/`
##### HTTP requests
| HTTP request | application/v2/tenant operation | Description |
| --- | --- | --- |
| GET |
List tenant information.
|
| | List tenants |
```
/application/v2/tenant/
```
Example response:
```json
[
"default"
]
```
|
| | Get tenant |
```
/application/v2/tenant/default
```
Example response:
```json
{
"message": "Tenant 'default' exists."
}
```
|
| PUT |
Create a new tenant.
|
| | Create tenant |
```
/application/v2/tenant/default
```
Response: A message with the name of the tenant created - example:
```json
{
"message" : "Tenant default created."
}
```
**Note:** This operation is asynchronous, it will eventually propagate to all config servers.
|
| DELETE |
Delete a tenant.
|
| | Delete tenant |
```
/application/v2/tenant/default
```
Response: A message with the deleted tenant:
```json
{
"message" : "Tenant default deleted."
}
```
**Note:** This operation is asynchronous, it will eventually propagate to all config servers.
|
##### Request parameters
None.
##### HTTP status codes
Non-exhaustive list of status codes. Any additional info is included in the body of the return call, JSON-formatted.
| Code | Description |
| --- | --- |
| 400 | Bad request. Client error. The error message should indicate the cause. |
| 404 | Not found. For example using a session id that does not exist. |
| 405 | Method not implemented. E.g. using GET where only POST or PUT is allowed. |
| 500 | Internal server error. Generic error. The error message should indicate the cause. |
##### Response format
Responses are in JSON format, with the following fields:
| Field | Description |
| --- | --- |
| message | An info/error message. |
Copyright © 2026 - [Cookie Preferences](#)
---
### /cluster/v2 API reference
[](/en/reference/api/cluster-v2.html.md "View as Markdown")
#### /cluster/v2 API reference
[](/en/reference/api/cluster-v2.html.md "View as Markdown")
The cluster controller has a /cluster/v2 API for viewing and modifying a content cluster state. To find the URL to access this API, identify the [cluster controller services](../../content/content-nodes.html#cluster-controller) in the application. Only the master cluster controller will be able to respond. The master cluster controller is the cluster controller alive that has the lowest index. Thus, one will typically use cluster controller 0, but if contacting it fails, try number 1 and so on. Using [vespa-model-inspect](/en/reference/operations/self-managed/tools.html#vespa-model-inspect):
```
$ vespa-model-inspect service -u container-clustercontroller
container-clustercontroller @ hostname.domain.com : admin
admin/cluster-controllers/0
http://hostname.domain.com:19050/ (STATE EXTERNAL QUERY HTTP)
http://hostname.domain.com:19117/ (EXTERNAL HTTP)
tcp/hostname.domain.com:19118 (MESSAGING RPC)
tcp/hostname.domain.com:19119 (ADMIN RPC)
```
In this example, there is only one clustercontroller, and the State Rest API is available on the port marked STATE and HTTP, 19050 in this example. This information can also be retrieved through the model config in the config server.
Find examples of API usage in [content nodes](../../content/content-nodes.html#cluster-v2-API-examples).
##### HTTP requests
| HTTP request | cluster/v2 operation | Description |
| --- | --- | --- |
| GET |
List cluster and nodes. Get cluster, node or disk states.
|
| | List content clusters |
```
/cluster/v2/
```
|
| | Get cluster state and list service types within cluster |
```
/cluster/v2/
```
|
| | List nodes per service type for cluster |
```
/cluster/v2//
```
|
| | Get node state |
```
/cluster/v2///
```
|
| PUT |
Set node state
|
| | Set node user state |
```
/cluster/v2///
```
|
##### Node state
Content and distributor nodes have state:
| State | Description |
| --- | --- |
|
`Up`
| The node is up and available to keep buckets and serve requests. |
|
`Down`
| The node is not available, and can not be used. |
|
`Stopping`
| This node is stopping and is expected to be down soon. This state is typically only exposed to the cluster controller to tell why the node stopped. The cluster controller will expose the node as down or in maintenance mode for the rest of the cluster. This state is thus not seen by the distribution algorithm. |
|
`Maintenance`
| This node is temporarily unavailable. The node is available for bucket placement, so redundancy is lower. Using this mode, new replicas of the documents stored on this node will not be created, allowing the node to be down with less of a performance impact on the rest of the cluster. This mode is typically used to mask a down state during controlled node restarts, or by an administrator that need to do some short maintenance work, like upgrading software or restart the node. |
|
`Retired`
| A retired node is available and serves requests. This state is used to remove nodes while keeping redundancy. Buckets are moved to other nodes (with low priority), until empty. Special considerations apply when using [grouped distribution](../../content/elasticity.html#grouped-distribution) as buckets are not necessarily removed. |
Distributor nodes start / transfer buckets quickly and are hence not in `maintenance` or `retired`.
Refer to [examples](../../content/content-nodes.html#cluster-v2-API-examples) of manipulating states.
##### Types
| Type | Spec | Description |
| --- | --- | --- |
|
cluster
| _\_ | The name given to a content cluster in a Vespa application. |
|
description
| _.\*_ | Description can contain anything that is valid JSON. However, as the information is presented in various interfaces, some which may present reasons for all the states in a cluster or similar, keeping it short and to the point makes it easier to fit the information neatly into a table and get a better cluster overview. |
|
group-spec
| _\_(\._\_)\* | The hierarchical group assignment of a given content node. This is a dot separated list of identifiers given in the application services.xml configuration. |
|
node
| [0-9]+ | The index or distribution key identifying a given node within the context of a content cluster and a service type. |
|
service-type
| (distributor|storage) | The type of the service to look at state for, within the context of a given content cluster. |
|
state-disk
| (up|down) | One of the valid disk states. |
|
state-unit
| [up](#up) | [stopping](#stopping) | [down](#down) |
The cluster controller fetches states from all nodes, called _unit states_. States reported from the nodes are either `up` or `stopping`. If the node can not be reached, a `down` state is assumed.
This means, the cluster controller detects failed nodes. The subsequent _generated states_ will have nodes in `down`, and the [ideal state algorithm](../../content/idealstate.html) will redistribute [buckets](../../content/buckets.html) of documents.
|
|
state-user
| [up](#up) | [down](#down) | [maintenance](#maintenance) | [retired](#retired) |
Use tools for [user state management](/en/operations/self-managed/admin-procedures.html#cluster-state).
- Retire a node from a cluster - use `retired` to move buckets to other nodes
- Short-lived maintenance work - use `maintenance` to avoid merging buckets to other nodes
- Fail a bad node. The cluster controller or an operator can set a node `down`
|
|
state-generated
| [up](#up) | [down](#down) | [maintenance](#maintenance) | [retired](#retired) |
The cluster controller generates the cluster state from the `unit` and `user` states, over time. The generated state is called the _cluster state_.
|
##### Request parameters
| Parameter | Type | Description |
| --- | --- | --- |
| recursive | number |
Number of levels, or `true` for all levels. Examples:
- Use `recursive=1` for a node request to also see all data
- use `recursive=2` to see all the node data within each service type
In recursive mode, you will see the same output as found in the spec below. However, where there is a `{ "link" : "" }` element, this element will be replaced by the content of that request, given a recursive value of one less than the request above.
|
##### HTTP status codes
Non-exhaustive list of status codes:
| Code | Description |
| --- | --- |
| 200 | OK. |
| 303 |
Cluster controller not master - master known.
This error means communicating with the wrong cluster controller. This returns a standard HTTP redirect, so the HTTP client can automatically redo the request on the correct cluster controller.
As the cluster controller available with the lowest index will be the master, the cluster controllers are normally queried in index order. Hence, it is unlikely to ever get this error, but rather fail to connect to the cluster controller if it is not the current master.
```
HTTP/1.1 303 See Other
Location: http://\/\Content-Type: application/json
{
"message" : "Cluster controllerindexnot master. Use master at indexindex.
}
```
|
| 503 |
Cluster controller not master - unknown or no master.
This error is used if the cluster controller asked is not master, and it doesn't know who the master is. This can happen, e.g. in a network split, where cluster controller 0 no longer can reach cluster controller 1 and 2, in which case cluster controller 0 knows it is not master, as it can't see the majority, and cluster controller 1 and 2 will vote 1 to master.
```
HTTP/1.1 503 Service Unavailable
Content-Type: application/json
{
"message" : "No known master cluster controller currently exist."
}
```
|
##### Response format
Responses are in JSON format, with the following fields:
| Field | Description |
| --- | --- |
| message | An error message — included for failed requests. |
| ToDo | Add more fields here. |
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [/cluster/v2 API reference](#page-title)
- [HTTP requests](#http-requests)
- [Node state](#node-state)
- [Types](#types)
- [Request parameters](#request-parameters)
- [HTTP status codes](#http-status-codes)
- [Response format](#response-format)
---
### Config API
[](/en/reference/api/config-v2.html.md "View as Markdown")
#### Config API
[](/en/reference/api/config-v2.html.md "View as Markdown")
Vespa provides a REST API for listing and retrieving config - alternatives are the [programmatic Java API](../../applications/configapi-dev.html#the-java-config-api). The Config API provides a way to inspect and retrieve all the config that can be generated by the config model for a given [tenant's active application](deploy-v2.html). Some, but not necessarily all, of those configs are used by services by [subscribing](../../applications/configapi-dev.html) to them.
The response format is JSON. The current API version is 2. All config servers provide the REST API. The API port is 19071 - use [vespa-model-inspect](../operations/self-managed/tools.html#vespa-model-inspect) service configserver to find config server hosts. Example: `http://myconfigserver.mydomain.com:19071/config/v2/tenant/msbe/application/articlesearch/`
The API is available after an application has been [deployed and activated](../../basics/applications.html#deploying-applications).
##### The application id
The API provides two ways to identify your application, given a tenant: one using only an application name, and one using application name, environment, region and instance. For the former, "short" form, a default environment, region and instance is used.
More formally, an _application id_ is a tuple of the form (_application_, _environment_, _region_, _instance_). The system currently provides shorthand to the id (_application_, "default", "default", "default").
Note: Multiple environments, regions and instances are not currently supported for application deployments, _default_ is always used.
Example URL using only application name: `http://myconfigserver.mydomain.com:19071/config/v2/tenant/media/application/articlesearch/media.config.server-list/clusters/0`
| Part | Description |
| --- | --- |
| media | Tenant |
| articlesearch | Application |
| media.config | Namespace of the requested config |
| server-list | Name of the requested config |
| clusters/0 | Config id of the requested config |
Example URL using full application id: `http://myconfigserver.mydomain.com:19071/config/v2/tenant/media/application/articlesearch/environment/test/region/us/instance/staging/media.config.server-list/clusters/0`
| Part | Description |
| --- | --- |
| media | Tenant |
| articlesearch | Name of the application |
| test | Environment |
| us | Region |
| staging | Instance |
| media.config | Namespace of the requested config |
| server-list | Name of the requested config |
| clusters/0 | Config id of the requested config |
In this API specification, the short form of the application id, i.e. only the application name, is used. The tenant `mytenant` and the application name `myapplication` is used throughout in examples.
##### GET /config/v2/tenant/mytenant/application/myapplication/
List the configs in the model, as [config id](../../applications/configapi-dev.html#config-id) specific URLs.
| Parameters |
| Parameter | Default | Description |
| --- | --- | --- |
| recursive | false | If true, include each config id in the model which produces the config, and list only the links to the config payload. If false, include the first level of the config ids in the listing of new list URLs, as explained above. |
|
| Request body | None |
| Response |
A list response includes two arrays:
- List-links to descend one level down in the config id hierarchy, named `children`.
- [Config payload](#payload) links for the current (top) level, named `configs`.
|
| Error Response |
N/A
|
Examples:
`GET /config/v2/tenant/mytenant/application/myapplication/`
```json
{
"children": [
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/config.sentinel/myconfigserver.mydomain.com/",
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/config.sentinel/hosts/",
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/config.model/admin/",
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/container.components/search/"
],
"configs": [
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/config.sentinel",
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/config.model",
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/container.components"
]
```
`GET /config/v2/tenant/mytenant/application/myapplication/?recursive=true`
```json
{
"configs": [
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/config.sentinel/myconfigserver.mydomain.com",
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/config.sentinel/hosts/myconfigserver.mydomain.com"
```
##### GET /config/v2/tenant/mytenant/application/myapplication/[namespace.name]/
| Parameters |
Same as above.
|
| Request body | None |
| Response |
List the configs in the model with the given namespace and name. List semantics as above.
|
| Error Response |
404 if the given namespace.name is not known to the config model.
|
Examples:
`GET /config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/`
```json
{
"children": [
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/search/",
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/clients/",
"http://myconfigserver.mydomain.com:19071/config/v1/vespaclient.config.feeder/docproc/"
]
"configs": [
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder",
]
}
```
`GET /config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/?recursive=true`
```json
{
"configs": [
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/search/qrsclusters/default",
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/clients/gateways",
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/clients/gateways/gateway",
```
##### GET /config/v2/tenant/mytenant/application/myapplication/[namespace.name]/[config/subid]/
| Parameters |
Same as above.
|
| Request body | None |
| Response |
List the configs in the model with the given namespace and name, and for which the given config id segment is a prefix.
|
| Error Response |
- 404 if the given namespace.name is not known to the config model.
- 404 if the given config id is not in the model.
|
Examples:
`GET /config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/search/`
```json
{
"children": [
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/search/qrsclusters/"
]
"configs": [
"http://myconfigserver.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/search"
]
}
```
`GET /config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/search/?recursive=true`
```json
{
"configs": [
"http://myhost.mydomain.com:19071/config/v2/tenant/mytenant/application/myapplication/vespaclient.config.feeder/search/qrsclusters/default"
]
}
```
##### GET /config/v2/tenant/mytenant/application/myapplication/[namespace.name]/[config/id]
| Parameters |
None
|
| Request body | None |
| Response |
Returns the config payload of the given `namespace.name/config/id`, formatted as JSON.
|
| Error Response |
Same as above.
|
Example:
`GET /config/v2/tenant/mytenant/application/myapplication/container.core.container-http/search/qrsclusters/default/qrserver.0`
```json
{
"enabled": "true",
"requestbuffersize": "65536",
"port": {
"search": "8080",
"host": ""
},
"fileserver": {
"throughsearch": "true"
}
}
```
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Config API](#page-title)
- [The application id](#application-id)
- [GET /config/v2/tenant/mytenant/application/myapplication/](#list-configs)
- [GET /config/v2/tenant/mytenant/application/myapplication/[namespace.name]/](#list-namespace)
- [GET /config/v2/tenant/mytenant/application/myapplication/[namespace.name]/[config/subid]/](#list-prefix)
- [GET /config/v2/tenant/mytenant/application/myapplication/[namespace.name]/[config/id]](#payload)
---
### Deploy API
[](/en/reference/api/deploy-v2.html.md "View as Markdown")
#### Deploy API
[](/en/reference/api/deploy-v2.html.md "View as Markdown")
This is the API specification and some examples for the HTTP Deploy API that can be used to deploy an application:
- [upload](#create-session)
- [prepare](#prepare-session)
- [activate](#activate-session)
The response format is JSON. Examples are found in the [use-cases](#use-cases). Also see the [deploy guide](/en/basics/applications.html#deploying-applications).
**Note:** To build a multi-application system, use one or three config server(s) per application. Best practise is using a [containerized](/en/operations/self-managed/docker-containers.html) architecture, also see [multinode-HA](https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA).
The current API version is 2. The API port is 19071 - use [vespa-model-inspect](/en/reference/operations/self-managed/tools.html#vespa-model-inspect) service configserver to find config server hosts. Example: `http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session`. Write operations return successfully after a majority of config servers have persisted changes (e.g. 2 out of 3 config servers).
Entities:
| session-id |
The session-id used in this API is generated by the server and is required for all operations after [creating](#create-session) a session. The session-id is valid if it is an active session, or it was created before [session lifetime](https://github.com/vespa-engine/vespa/blob/master/configdefinitions/src/vespa/configserver.def) has expired, the default value being 1 hour.
|
| path |
An application file path in a request URL or parameter refers to a relative path in the application package. A path ending with "/" refers to a directory.
|
Use [Vespa CLI](../../clients/vespa-cli.html) to deploy from the command line.
##### POST /application/v2/tenant/default/prepareandactivate
Creates a new session with the application package that is included in the request, prepares it and then activates it. See details in the steps later in this document
| Parameters |
| Name | Default | Description |
| --- | --- | --- |
| | | |
|
| Request body |
| Required | Content | Note |
| --- | --- | --- |
| Yes | A compressed [application package](../applications/application-packages.html) (with gzip or zip compression) | Set `Content-Type` HTTP header to `application/x-gzip` or `application/zip`. |
|
| Response |
See [active](#activate-session).
|
Example:
```
$ (cd src/main/application && zip -r - .) | \
curl --header Content-Type:application/zip --data-binary @- \
localhost:19071/application/v2/tenant/default/prepareandactivate
```
```json
{
"log": [
{
"time": 1619448107299,
"level": "WARNING",
"message": "Host named 'vespa-container' may not receive any config since it is not a canonical hostname. Disregard this warning when testing in a Docker container."
}
],
"tenant": "default",
"session-id": "3",
"url": "http://localhost:19071/application/v2/tenant/default/application/default/environment/prod/region/default/instance/default",
"message": "Session 3 for tenant 'default' prepared and activated.",
"configChangeActions": {
"restart": [],
"refeed": [],
"reindex": []
}
}
```
##### POST /application/v2/tenant/default/session
Creates a new session with the application package that is included in the request.
| Parameters |
| Name | Default | Description |
| --- | --- | --- |
| from | N/A | Use when you want to create a new session based on an active application. The value supplied should be a URL to an active application. |
|
| Request body |
| Required | Content | Note |
| --- | --- | --- |
| Yes, unless `from` parameter is used | A compressed [application package](../applications/application-packages.html) (with gzip or zip compression) | It is required to set the `Content-Type` HTTP header to `application/x-gzip` or `application/zip`, unless the `from` parameter is used. |
|
| Response | The response contains:
- A [session-id](#session-id) to the application that was created.
- A [prepared](#prepare-session) URL for preparing the application.
|
Examples (both requests return the same response):
- `POST /application/v2/tenant/default/session`
- `POST /application/v2/tenant/default/session?from=http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/application/default/environment/default/region/default/instance/default`
```json
{
"tenant": "default",
"session-id": "1",
"prepared": "http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/session-id/prepared/",
"content": "http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/session-id/content/",
"message": "Session 1 for tenant 'default' created."
}
```
##### PUT /application/v2/tenant/default/session/[[session-id](#session-id)]/content/[[path](#path)]
Writes the content to the given path, or creates a directory if the path ends with '/'.
| Parameters | None |
| Request body |
- If path is a directory, none.
- If path is a file, the contents of the file.
|
| Response |
None
- Any errors or warnings from writing the file/creating the directory.
|
##### GET /application/v2/tenant/default/session/[[session-id](#session-id)]/content/[[path](#path)]
Returns the content of the file at this path, or lists files and directories if `path` ends with '/'.
| Parameters |
| Name | Default | Description |
| --- | --- | --- |
| recursive | false | If _true_, directory content will be listed recursively. |
| return | content |
- If set to content and path refers to a file, the content will be returned.
- If set to content and path refers to a directory, the files and subdirectories in the directory will be listed.
- If set to status and path refers to a file, the file status and hash will be returned.
- If set to status and path refers to a directory, a list of file/subdirectory statuses and hashes will be returned.
|
|
| Request body | None. |
| Response |
- If path is a directory: a JSON array of URLs to the files and subdirectories of that directory.
- If path is a file: the contents of the file.
- If status parameter is set, the status and hash will be returned.
|
Examples:
`GET /application/v2/tenant/default/session/3/content/`
```json
[
"http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/hosts.xml",
"http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/services.xml",
"http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/schemas/"
]
```
`GET /application/v2/tenant/default/session/3/content/?recursive=true`
```json
[
"http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/hosts.xml",
"http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/services.xml",
"http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/schemas/",
"http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/schemas/music.sd",
"http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/schemas/video.sd"
]
```
`GET /application/v2/tenant/default/session/3/content/hosts.xml`
```xml
vespa1
vespa2
```
`GET /application/v2/tenant/default/session/3/content/hosts.xml?return=status`
```json
{
"name": "http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/hosts.xml",
"status": "new",
"md5": "03d7cff861fcc2d88db70b7857d4d452"
}
```
`GET /application/v2/tenant/default/session/3/content/schemas/?return=status`
```json
[
{
"name": "http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/schemas/music.sd",
"status": "new",
"md5": "03d7cff861fcc2d88db70b7857d4d452"
},
{
"name": "http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/schemas/video.sd",
"status": "changed",
"md5": "03d7cff861fcc2d88db70b7857d4d452"
},
{
"name": "http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/content/schemas/book.sd",
"status": "deleted",
"md5": "03d7cff861fcc2d88db70b7857d4d452"
}
]
```
##### DELETE /application/v2/tenant/default/session/[[session-id](#session-id)]/content/[[path](#path)]
Deletes the resource at the given path.
| Parameters | None |
| Request body |
None
|
| Response |
Any errors or warnings from deleting the resource.
|
##### PUT /application/v2/tenant/default/session/[[session-id](#session-id)]/prepared
Prepares an application with the [session-id](#session-id) given.
| Parameters |
| Parameter | Default | Description |
| --- | --- | --- |
| applicationName | N/A | Name of the application to be deployed |
| environment | default | Environment where application should be deployed |
| region | default | Region where application should be deployed |
| instance | default | Name of application instance |
| debug | false | If true, include stack trace in response if prepare fails. |
| timeout | 360 seconds | Timeout in seconds to wait for session to be prepared. |
|
| Request body |
None
|
| Response |
Returns a [session-id](#session-id) and a link to activate the session.
- Log with any errors or warnings from preparing the application.
- An [activate](#activate-session) URL for activating the application with this [session-id](#session-id), if there were no errors.
- A list of actions (possibly empty) that must be performed in order to apply some config changes between the current active application and this next prepared application. These actions are organized into three categories; _restart_, _reindex_, and _refeed_:
- _Restart_ actions are done after the application has been activated and are handled by restarting all listed services. See [schemas](../schemas/schemas.html#modifying-schemas) for details.
- _Reindex_ actions are special refeed actions that Vespa [handles automatically](../../operations/reindexing.html), if the [reindex](#reindex) endpoint below is used.
- _Refeed_ actions require several steps to handle. See [schemas](../schemas/schemas.html#modifying-schemas) for details.
|
Example:
`PUT /application/v2/tenant/default/session/3/prepared`
```json
{
"tenant": "default",
"session-id": "3",
"activate": "http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/session/3/active",
"message": "Session 3 for tenant 'default' prepared.",
"log": [
{ "level": "WARNING",
"message": "Warning message 1",
"time": 1430134091319
},
{ "level": "WARNING",
"message": "Warning message 2",
"time": 1430134091320
}
],
"configChangeActions": {
"restart": [ {
"clusterName": "mycluster",
"clusterType": "search",
"serviceType": "searchnode",
"messages": ["Document type 'test': Field 'f1' changed: add attribute aspect"],
"services": [ {
"serviceName": "searchnode",
"serviceType": "searchnode",
"configId": "mycluster/search/cluster.mycluster/0",
"hostName": "myhost.mydomain.com"
} ]
} ],
"reindex": [ {
"documentType": "test",
"clusterName": "mycluster",
"messages": ["Document type 'test': Field 'f1' changed: add index aspect"],
"services": [ {
"serviceName": "searchnode",
"serviceType": "searchnode",
"configId": "mycluster/search/cluster.mycluster/0",
"hostName": "myhost.mydomain.com"
} ]
} ]
}
}
```
##### GET /application/v2/tenant/default/session/[[session-id](#session-id)]/prepared
Returns the state of a prepared session. The response is the same as a successful [prepare](#prepare-session) operation (above), however the _configChangeActions_ element will be empty.
##### PUT /application/v2/tenant/default/session/[[session-id](#session-id)]/active
Activates an application with the [session-id](#session-id) given. The [session-id](#session-id) must be for a [prepared session](#prepare-session). The operation will make sure the session is activated on all config servers.
| Parameters |
| Parameter | Default | Description |
| --- | --- | --- |
| timeout | 60 seconds | Timeout in seconds to wait for session to be activated (when several config servers are used, they might need to sync before activate can be done). |
|
| Request body | None |
| Response |
Returns a [session-id](#session-id), a message and a URL to the activated application.
- [session-id](#session-id)
- Message
|
Example:
`PUT /application/v2/tenant/default/session/3/active`
```json
{
"tenant": "default",
"session-id": "3",
"message": "Session 3 for tenant 'default' activated.",
"url": "http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/application/default/environment/default/region/default/instance/default"
}
```
##### GET /application/v2/tenant/default/application/
Returns a list of the currently active applications for the given tenant.
| Parameters | None |
| Request body | None |
| Response |
Returns a list of applications
- Array of active applications
|
Example:
`GET /application/v2/tenant/default/application/`
```json
{
["http://myconfigserver.mydomain.com:19071/application/v2/tenant/default/application/default/environment/default/region/default/instance/default"]
}
```
##### GET /application/v2/tenant/default/application/default
Gets info about the application.
| Parameters | None |
| Request body | None |
| Response |
Returns information about the application specified.
- config generation
|
Example:
`GET /application/v2/tenant/default/application/default`
```json
{
"generation": 2
}
```
##### GET /application/v2/tenant/default/application/default/environment/default/region/default/instance/default/reindexing
Returns [reindexing](../../operations/reindexing.html) status for the given application.
| Parameters | N/A |
| Request body | N/A |
| Response | JSON detailing current reindexing status for the application, with all its clusters and document types.
- Status for each content cluster in the application, by name:
- Status of each document type in the cluster, by name:
- Last time reindexing was triggered for this document type.
- Current status of reindexing.
- Optional start time of reindexing.
- Optional end time of reindexing.
- Optional progress of reindexing, from 0 to 1.
- Pseudo-speed of reindexing.
|
Example:
`GET /application/v2/tenant/default/application/default/environment/default/region/default/instance/default/reindexing`
```json
{
"clusters": {
"db": {
"ready": {
"test_artifact": {
"readyMillis": 1607937250998,
"startedMillis": 1607940060012,
"state": "running",
"speed": 1.0,
"progress": 0.04013824462890625
},
"test_result": {
"readyMillis": 1607688477294,
"startedMillis": 1607690520026,
"endedMillis": 1607709294236,
"speed": 0.1,
"state": "successful"
},
"test_run": {
"readyMillis": 1607937250998,
"state": "pending"
}
}
}
}
}
```
##### POST /application/v2/tenant/default/application/default/environment/default/region/default/instance/default/reindex
Marks specified document types in specified clusters of an application as ready for [reindexing](../../operations/reindexing.html). Reindexing itself starts with the next redeployment of the application. To stop an ongoing reindexing, see [updating reindexing](#update-reindexing) below. All document types in all clusters are reindexed unless restricted, using parameters as specified:
| Parameters |
| Name | Description |
| --- | --- |
| clusterId | A comma-separated list of content clusters to limit reindexing to. All clusters are reindexed if this is not present. |
| documentType | A comma-separated list of document types to limit reindexing to. All document types are reindexed if this is not present. |
| indexedOnly | Boolean: whether to mark reindexing ready only for document types with indexing mode _index_ and at least one field with the indexing statement `index`. Default is `false`. |
| speed | Number (0–10], default 1: Indexing pseudo speed - balance speed vs. resource use. Example: speed=0.1 |
|
| Request body | N/A |
| Response | A human-readable message indicating what reindexing was marked as ready. |
Example:
`POST /application/v2/tenant/default/application/default/environment/default/region/default/instance/default/reindex?clusterId=foo,bar&documentType=moo,baz&indexedOnly=true`
```json
{
"message": "Reindexing document types [moo, baz] in 'foo', [moo] in 'bar' of application default.default"
}
```
##### PUT /application/v2/tenant/default/application/default/environment/default/region/default/instance/default/reindex
Modifies [reindexing](../../operations/reindexing.html) of specified document types in specified clusters of an application. Specifically, this can be used to alter the pseudo-speed of the reindexing, optionally halting it by specifying a speed of `0`; reindexing for the specified types will remain dormant until either speed is increased again, or a new reindexing is triggered (see [trigger reindexing](#reindex)). Speed changes become effective with the next redeployment of the application. Reindexing for all document types in all clusters are affected if no other parameters are specified:
| Parameters |
| Name | Description |
| --- | --- |
| clusterId | A comma-separated list of content clusters to limit the changes to. Reindexing for all clusters are modified if this is not present. |
| documentType | A comma-separated list of document types to limit the changes to. Reindexing for all document types are modified if this is not present. |
| indexedOnly | Boolean: whether to modify reindexing only for document types with indexing mode _index_ and at least one field with the indexing statement `index`. Default is `false`. |
| speed | Number [0–10], required: Indexing pseudo speed - balance speed vs. resource use. Example: speed=0.1 |
|
| Request body | N/A |
| Response | A human-readable message indicating what reindexing was modified. |
Example:
`PUT /application/v2/tenant/default/application/default/environment/default/region/default/instance/default/reindex?clusterId=foo,bar&documentType=moo,baz&speed=0.618`
```json
{
"message": "Set reindexing speed to '0.618' for document types [moo, baz] in 'foo', [moo] in 'bar' of application default.default"
}
```
##### GET /application/v2/tenant/default/application/default/environment/default/region/default/instance/default/content/[[path](#path)]
Returns content at the given path for an application. See [getting content](#content-get) for usage and response.
##### DELETE /application/v2/tenant/default/application/default
Deletes an active application.
| Parameters | None |
| Request body | None |
| Response |
Returns a message stating if the operation was successful or not
|
Example:
`DELETE /application/v2/tenant/default/application/default`
```json
{
"message": "Application 'default' was deleted"
}
```
##### GET /application/v2/host/[hostname]
Gets information about which tenant and application a hostname is used by.
| Parameters | None |
| Request body | None |
| Response |
Returns a message with tenant and application details.
|
Example:
`GET /application/v2/host/myhost.mydomain.com`
```json
{
"tenant": "default"
"application": "default"
"environment": "default"
"region": "default"
"instance": "default"
}
```
##### Error Handling
Errors are returned using standard HTTP status codes. Any additional info is included in the body of the return call, JSON-formatted. The general format for an error response is:
```json
{
"error-code": "ERROR_CODE",
"message": "An error message"
}
```
| HTTP status code | Error code | Description |
| --- | --- | --- |
| 400 | BAD\_REQUEST | Bad request. Client error. The error message should indicate the cause. |
| 400 | INVALID\_APPLICATION\_PACKAGE | There is an error in the application package. The error message should indicate the cause. |
| 400 | OUT\_OF\_CAPACITY | Not enough nodes available for the request to be fulfilled. |
| 401 | | Not authorized. The error message should indicate the cause. |
| 404 | NOT\_FOUND | Not found. E.g. when using a session-id that doesn't exist. |
| 405 | METHOD\_NOT\_ALLOWED | Method not implemented. E.g. using GET where only POST or PUT is allowed. |
| 409 | ACTIVATION\_CONFLICT | Conflict, returned when activating an application fails due to a conflict with other changes to the same application (in another session). Client should retry. |
| 500 | INTERNAL\_SERVER\_ERROR | Internal server error. Generic error. The error message should indicate the cause. |
##### Access log
Requests are logged in the [access log](../../operations/access-logging.html) which can be found at _$VESPA\_HOME/logs/vespa/configserver/access-json.log_, example:
```json
{
"ip": "172.17.0.2",
"time": 1655665104.751,
"duration": 1.581,
"responsesize": 230,
"requestsize": 0,
"code": 200,
"method": "PUT",
"uri": "/application/v2/tenant/default/session/2/prepared",
"version": "HTTP/2.0",
"agent": "vespa-deploy",
"host": "b614c9ff04d7:19071",
"scheme": "https",
"localport": 19071,
"peeraddr": "172.17.0.2",
"peerport": 47480,
"attributes": {
"http2-stream-id":"1"
}
}
```
##### Use Cases
It is assumed that the tenant _default_ is already created in these use cases, and the application package is in _app_.
###### Create, prepare and activate an application
Create a session with the application package:
```
$ (cd app && zip -r - .) | \
curl -s --header Content-Type:application/zip --data-binary @- \
"http://host:19071/application/v2/tenant/default/session"
```
Prepare the application with the URL in the _prepared_ link from the response:
```
$ curl -s -X PUT "http://host:19071/application/v2/tenant/default/session/1/prepared?applicationName=default"
```
Activate the application with the URL in the _activate_ link from the response:
```
$ curl -s -X PUT "http://host:19071/application/v2/tenant/default/session/1/active"
```
###### Modify the application package
Dump _services.xml_ from session 1:
```
$ curl -s -X GET "http://host:19071/application/v2/tenant/default/session/1/content/services.xml"
```
```xml
12345
```
Session 1 is activated and cannot be changed - create a new session based on the active session:
```
$ curl -s -X POST "http://host:19071/application/v2/tenant/default/session?from=http://host:19071/application/v2/tenant/default/application/default/environment/default/region/default/instance/default"
```
Modify rpcport to 12346 in _services.xml_, deploy the change:
```
$ curl -s -X PUT --data-binary @app/services.xml \
"http://host:19071/application/v2/tenant/default/session/2/content/services.xml"
```
Get _services.xml_ from session 2 to validate:
```
$ curl -s -X GET "http://host:19071/application/v2/tenant/default/session/2/content/services.xml"
```
```xml
12346
```
To add the file _files/test1.txt_, first create the directory, then add the file:
```
$ curl -s -X PUT "http://host:19071/application/v2/tenant/default/session/2/content/files/"
$ curl -s -X PUT --data-binary @app/files/test1.txt \
"http://host:19071/application/v2/tenant/default/session/2/content/files/test1.txt"
```
Prepare and activate the session:
```
$ curl -s -X PUT "http://host:19071/application/v2/tenant/default/session/2/prepared?applicationName=fooapp"
$ curl -s -X PUT "http://host:19071/application/v2/tenant/default/session/2/active"
```
###### Rollback
If you need to roll back to a previous version of the application package this can be achieved by creating a new session based on the previous known working version by passing the corresponding session-id in the _from_ argument, see [creating a session](#create-session)
Also see [rollback](/en/applications/deployment.html#rollback).
Copyright © 2026 - [Cookie Preferences](#)
###### On this page:
- [Deploy API](#page-title)
- [POST /application/v2/tenant/default/prepareandactivate](#prepareandactivate)
- [POST /application/v2/tenant/default/session](#create-session)
- [PUT /application/v2/tenant/default/session/[](#content-put)
- [GET /application/v2/tenant/default/session/[](#content-get)
- [DELETE /application/v2/tenant/default/session/[](#content-delete)
- [PUT /application/v2/tenant/default/session/[](#prepare-session)
- [GET /application/v2/tenant/default/session/[](#get-prepare-session)
- [PUT /application/v2/tenant/default/session/[](#activate-session)
- [GET /application/v2/tenant/default/application/](#get-application)
- [GET /application/v2/tenant/default/application/default](#get-application-info)
- [GET /application/v2/tenant/default/application/default/environment/default/region/default/instance/default/reindexing](#reindexing)
- [POST /application/v2/tenant/default/application/default/environment/default/region/default/instance/default/reindex](#reindex)
- [PUT /application/v2/tenant/default/application/default/environment/default/region/default/instance/default/reindex](#update-reindexing)
- [GET /application/v2/tenant/default/application/default/environment/default/region/default/instance/default/content/[](#get-application-content)
- [DELETE /application/v2/tenant/default/application/default](#delete-application)
- [GET /application/v2/host/[hostname]](#get-host-info)
- [Error Handling](#error-handling)
- [Access log](#access-log)
- [Use Cases](#use-cases)
- [Create, prepare and activate an application](#use-case-start)
- [Modify the application package](#use-case-modify)
- [Rollback](#rollback)
---
### /document/v1 API reference
[](/en/reference/api/document-v1.html.md "View as Markdown")
#### /document/v1 API reference
[](/en/reference/api/document-v1.html.md "View as Markdown")
This is the /document/v1 API reference documentation. Use this API for synchronous [Document](../../schemas/documents.html) operations to a Vespa endpoint - refer to [reads and writes](../../writing/reads-and-writes.html) for other options.
The [document/v1 API guide](../../writing/document-v1-api-guide.html) has examples and use cases.
**Note:** Mapping from document IDs to /document/v1/ URLs is found in [Document IDs](../../schemas/documents.html#id-scheme) - also see [troubleshooting](../../writing/document-v1-api-guide.html#troubleshooting).
Some examples use _number_ and _group_[document ID](../../schemas/documents.html#document-ids) modifiers. These are special cases that only work as expected for document types with [mode=streaming or mode=store-only](../applications/services/content.html#document). Do not use group or number modifiers with regular indexed mode document types.
##### Configuration
To enable the API, add `document-api` in the serving container cluster - [services.xml](../applications/services/container.html):
```xml
```
##### HTTP requests
| HTTP request | document/v1 operation | Description |
| --- | --- | --- |
| GET |
_Get_ a document by ID or _Visit_ a set of documents by selection.
|
| | Get | Get a document:
```
/document/v1///docid/
/document/v1///number//
/document/v1///group//
```
Optional parameters:
- [cluster](#cluster)
- [fieldSet](#fieldset)
- [timeout](#timeout)
- [tracelevel](#tracelevel)
|
| | Visit |
Iterate over and get all documents, or a [selection](#selection) of documents, in chunks, using [continuation](#continuation) tokens to track progress. Visits are a linear scan over the documents in the cluster.
```
/document/v1/
```
It is possible to specify namespace and document type with the visit path:
```
/document/v1///docid
```
Documents can be grouped to limit accesses to a subset. A group is defined by a numeric ID or string — see [id scheme](../../schemas/documents.html#id-scheme).
```
/document/v1///group/
/document/v1///number/
```
Mandatory parameters:
- [cluster](#cluster) - Visits can only retrieve data from _one_ content cluster, so `cluster` **must** be specified for requests at the root `/document/v1/` level, or when there is ambiguity. This is required even if the application has only one content cluster.
Optional parameters:
- [bucketSpace](#bucketspace) - Parent documents are [global](../applications/services/content.html#document) and in the `global` [bucket space](#bucketspace). By default, visit will visit non-global documents in the `default` bucket space, unless document type is indicated, and is a global document type.
- [concurrency](#concurrency) - Use to configure backend parallelism for each visit HTTP request.
- [continuation](#continuation)
- [fieldSet](#fieldset)
- [selection](#selection)
- [sliceId](#sliceid)
- [slices](#slices) - Split visiting of the document corpus across more than one HTTP request—thus allowing the concurrent use of more HTTP containers—use the `slices` and `sliceId` parameters.
- [stream](#stream) - It's recommended enabling streamed HTTP responses, with the [stream](#stream) parameter, as this reduces memory consumption and reduces HTTP overhead.
- [timeout](#timeout)
- [tracelevel](#tracelevel)
- [wantedDocumentCount](#wanteddocumentcount)
- [fromTimestamp](#fromtimestamp)
- [toTimestamp](#totimestamp)
- [includeRemoves](#includeRemoves)
Optional request headers:
- [Accept](#accept) - specify the desired response format.
|
| POST |
_Put_ a given document, by ID, or _Copy_ a set of documents by selection from one content cluster to another.
|
| | Put | Write the document contained in the request body in JSON format.
```
/document/v1///docid/
/document/v1///group/
/document/v1///number/
```
Optional parameters:
- [condition](#condition) - Use for conditional writes.
- [route](#route)
- [timeout](#timeout)
- [tracelevel](#tracelevel)
|
| | Copy |
Write documents visited in source [cluster](#cluster) to the [destinationCluster](#destinationcluster) in the same application. A [selection](#selection) is mandatory — typically the document type. Supported paths (see [visit](#visit) above for semantics):
```
/document/v1/
/document/v1///docid/
/document/v1///group/
/document/v1///number/
```
Mandatory parameters:
- [cluster](#cluster)
- [destinationCluster](#destinationcluster)
- [selection](#selection)
Optional parameters:
- [bucketSpace](#bucketspace)
- [continuation](#continuation)
- [timeChunk](#timechunk)
- [timeout](#timeout)
- [tracelevel](#tracelevel)
|
| PUT |
_Update_ a document with the given partial update, by ID, or _Update where_ the given selection is true.
|
| | Update | Update a document with the partial update contained in the request body in the [document update JSON format](../schemas/document-json-format.html#update).
```
/document/v1///docid/
```
Optional parameters:
- [condition](#condition) - use for conditional writes
- [create](#create) - use to create empty documents when updating non-existent ones.
- [route](#route)
- [timeout](#timeout)
- [tracelevel](#tracelevel)
|
| | Update where |
Update visited documents in [cluster](#cluster) with the partial update contained in the request body in the [document update JSON format](../schemas/document-json-format.html#update). Supported paths (see [visit](#visit) above for semantics):
```
/document/v1///docid/
/document/v1///group/
/document/v1///number/
```
Mandatory parameters:
- [cluster](#cluster)
- [selection](#selection)
Optional parameters:
- [bucketSpace](#bucketspace) - See [visit](#visit), `default` or `global` bucket space
- [continuation](#continuation)
- [stream](#stream)
- [timeChunk](#timechunk)
- [timeout](#timeout)
- [tracelevel](#tracelevel)
|
| DELETE |
_Remove_ a document, by ID, or _Remove where_ the given selection is true.
|
| | Remove | Remove a document.
```
/document/v1///docid/
```
Optional parameters:
- [condition](#condition)
- [route](#route)
- [timeout](#timeout)
- [tracelevel](#tracelevel)
|
| | Delete where |
Delete visited documents from [cluster](#cluster). Supported paths (see [visit](#visit) above for semantics):
```
/document/v1/
/document/v1///docid/
/document/v1///group/
/document/v1///number/
```
Mandatory parameters:
- [cluster](#cluster)
- [selection](#selection)
Optional parameters:
- [bucketSpace](#bucketspace) - See [visit](#visit), `default` or `global` bucket space
- [continuation](#continuation)
- [stream](#stream)
- [timeChunk](#timechunk)
- [timeout](#timeout)
- [tracelevel](#tracelevel)
|
##### Request parameters
| Parameter | Type | Description |
| --- | --- | --- |
| bucketSpace | String |
Specify the bucket space to visit. Document types marked as `global` exist in a separate _bucket space_ from non-global document types. When visiting a particular document type, the bucket space is automatically deduced based on the provided type name. When visiting at a root `/document/v1/` level this information is not available, and the non-global ("default") bucket space is visited by default. Specify `global` to visit global documents instead. Supported values: `default` (for non-global documents) and `global`.
|
| cluster | String |
Name of [content cluster](../../content/content-nodes.html) to GET from, or visit.
|
| concurrency | Integer |
Sends the given number of visitors in parallel to the backend, improving throughput at the cost of resource usage. Default is 1. When `stream=true`, concurrency limits the maximum concurrency, which is otherwise unbounded, but controlled by a dynamic throttle policy.
**Important:** Given a concurrency parameter of _N_, the worst case for memory used while processing the request grows linearly with _N_, unless [stream](#stream) mode is turned on. This is because the container currently buffers all response data in memory before sending them to the client, and all sent visitors must complete before the response can be sent.
|
| condition | String |
For test-and-set. Run a document operation conditionally — if the condition fails, a _412 Precondition Failed_ is returned. See [example](../../writing/document-v1-api-guide.html#conditional-writes).
|
| continuation | String |
When visiting, a continuation token is returned as the `"continuation"` field in the JSON response, as long as more documents remain. Use this token as the `continuation` parameter to visit the next chunk of documents. See [example](../../writing/document-v1-api-guide.html#data-dump).
|
| create | Boolean |
If `true`, updates to non-existent documents will create an empty document to update. See [create if nonexistent](../../writing/document-v1-api-guide.html#create-if-nonexistent).
|
| destinationCluster | String |
Name of [content cluster](../../content/content-nodes.html) to copy to, during a copy visit.
|
| dryRun | Boolean |
Used by the [vespa-feed-client](../../clients/vespa-feed-client.html) using `--speed-test` for bandwidth testing, by setting to `true`.
|
| fieldSet | String |
A [field set string](../../schemas/documents.html#fieldsets) with the set of document fields to fetch from the backend. Default is the special `[document]` fieldset, returning all _document_ fields. To fetch specific fields, use the name of the document type, followed by a comma-separated list of fields (for example `music:artist,song` to fetch two fields declared in `music.sd`).
|
| route | String |
The route for single document operations, and for operations generated by [copy](#copy), [update](#update-where) or [deletion](#delete-where) visits. Default value is `default`. See [routes](../../writing/document-routing.html).
|
| selection | String |
Select only a subset of documents when [visiting](../../writing/visiting.html) — details in [document selector language](../writing/document-selector-language.html).
|
| sliceId | Integer |
The slice number of the visit represented by this HTTP request. This number must be non-negative and less than the number of [slices](#slices) specified for the visit - e.g., if the number of slices is 10, `sliceId` is in the range [0-9].
**Note:** If the number of distribution bits change during a sliced visit, the results are undefined. Thankfully, this is a very rare occurrence and is only triggered when adding content nodes.
|
| slices | Integer |
Split the document corpus into this number of independent slices. This lets multiple, concurrent series of HTTP requests advance the same logical visit independently, by specifying a different [sliceId](#sliceid) for each.
|
| stream | Boolean |
Whether to stream the HTTP response, allowing data to flow as soon as documents arrive from the backend. This obsoletes the [wantedDocumentCount](#wanteddocumentcount) parameter. The HTTP status code will always be 200 if the visit is successfully initiated. Default value is false.
|
| format.tensors | String |
Controls how tensors are rendered in the result.
| Value | Description |
| --- | --- |
| `short` | **Default**. Render the tensor value in an object having two keys, "type" containing the value, and "cells"/"blocks"/"values" ([depending on the type](../schemas/document-json-format.html#tensor)) containing the tensor content.
Render the tensor content in the [type-appropriate short form](../schemas/document-json-format.html#tensor). |
| `long` | Render the tensor value in an object having two keys, "type" containing the value, and "cells" containing the tensor content.
Render the tensor content in the [general verbose form](../schemas/document-json-format.html#tensor). |
| `short-value` | Render the tensor content directly.
Render the tensor content in the [type-appropriate short form](../schemas/document-json-format.html#tensor). |
| `long-value` | Render the tensor content directly.
Render the tensor content in the [general verbose form](../schemas/document-json-format.html#tensor). |
|
| timeChunk | String |
Target time to spend on one chunk of a copy, update or remove visit; with optional ks, s, ms or µs unit. Default value is 60.
|
| timeout | String |
Request timeout in seconds, or with optional ks, s, ms or µs unit. Default value is 180s.
|
| tracelevel | Integer |
Number in the range [0,9], where higher gives more details. The trace dumps which nodes and chains the document operation has touched. See [routes](../../writing/document-routing.html).
|
| wantedDocumentCount | Integer |
Best effort attempt to not respond to the client before `wantedDocumentCount` number of documents have been visited. Response may still contain fewer documents if there are not enough matching documents left to visit in the cluster, or if the visiting times out. This parameter is intended for the case when you have relatively few documents in your cluster and where each visit request would otherwise process only a handful of documents.
The maximum value of `wantedDocumentCount` is bounded by an implementation-specific limit to prevent excessive resource usage. If the cluster has many documents (on the order of tens of millions), there is no need to set this value.
|
| fromTimestamp | Integer |
Filters the returned document set to only include documents that were last modified at a time point equal to or higher to the specified value, in microseconds from UTC epoch. Default value is 0 (include all documents).
|
| toTimestamp | Integer |
Filters the returned document set to only include documents that were last modified at a time point lower than the specified value, in microseconds from UTC epoch. Default value is 0 (sentinel value; include all documents). If non-zero, must be greater than, or equal to, `fromTimestamp`.
|
| includeRemoves | Boolean |
Include recently removed document IDs, along with the set of returned documents. By default, only documents currently present in the corpus are returned in the `"documents"` array of the response; when this parameter is set to `"true"`, documents that were recently removed, and whose tombstones still exist, are also included in that array, as entries on the form `{ "remove": "id:ns:type::foobar" }`. See [here](/en/operations/self-managed/admin-procedures.html#data-retention-vs-size) for specifics on tombstones, including their lifetime.
|
##### HTTP request headers
| Header | Values | Description |
| --- | --- | --- |
| Accept | `application/json` or `application/jsonl` |
The [Accept](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept) header lets the client specify to the server what [media (MIME) types](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/MIME_types) it accepts as the response format.
All Document V1 API calls support `application/json` for returning [JSON](#json) responses. [Streaming visiting](#stream) additionally supports `application/jsonl` for returning [JSON Lines](#json-lines) (JSONL) since Vespa 8.593.
To ensure compatibility with older versions, make sure to check the `Content-Type`[HTTP response header](#http-response-headers). A JSONL response will always have a `Content-Type` media type of `application/jsonl`, and JSON wil always have a media type of `application/json`.
Multiple acceptable types can be specified. JSONL will be returned if (and only if) `application/jsonl` is part of the list _and_ no other media types have a higher [quality value](https://httpwg.org/specs/rfc9110.html#quality.values).
Example:
```
Accept: application/jsonl
```
If the client accepts both JSON and JSONL, the server will respond with JSONL:
```
Accept: application/json, application/jsonl
```
For backwards compatibility, if no `Accept` header is provided (or if no provided media types are acceptable) `application/json` is assumed.
|
##### Request body
POST and PUT requests must include a body for single document operations; PUT must also include a body for [update where](#update-where) visits. A field has a _value_ for a POST and an _update operation object_ for PUT. Documents and operations use the [document JSON format](../schemas/document-json-format.html). The document fields must match the [schema](../../basics/schemas.html):
```json
{
"fields": {
"": ""
}
}
```
```json
{
"fields": {
"": {
"" : ""
}
}
}
```
The _update-operation_ is most often `assign` - see [update operations](../schemas/document-json-format.html#update-operations) for the full list. Values for `id` / `put` / `update` in the request body are silently dropped. The ID is generated from the request path, regardless of request body data - example:
```json
{
"put" : "id:mynamespace:music::123",
"fields": {
"title": "Best of"
}
}
```
This makes it easier to generate a feed file that can be used for both the [vespa-feed-client](../../clients/vespa-feed-client.html) and this API.
##### HTTP status codes
| Code | Description |
| --- | --- |
| 200 | OK. Attempts to remove or update a non-existent document also yield this status code (see 412 below). |
| 204 | No Content. Successful response to OPTIONS request. |
| 400 | Bad request. Returned for undefined document types + other request errors. See [13465](https://github.com/vespa-engine/vespa/issues/13465) for defined document types not assigned to a content cluster when using PUT. Inspect `message` for details. |
| 404 | Not found; the document was not found. This is only used when getting documents. |
| 405 | Method Not Allowed. HTTP method is not supported by the endpoint. Valid combinations are listed [above](#http-requests) |
| 412 | [condition](#condition) is not met. Inspect `message` for details. This is also the result when a condition if specified, but the document does not exist. |
| 413 | Content too large; used for POST and PUT requests that are above the [request size limit](../../writing/document-v1-api-guide.html#request-size-limit). |
| 429 | Too many requests; the document API has too many inflight feed operations, retry later. |
| 500 | Server error; an unspecified error occurred when processing the request/response. |
| 503 | Service unavailable; the document API was unable to produce a response at this time. |
| 504 | Gateway timeout; the document API failed to respond within the given (or default 180s) timeout. |
| 507 | Insufficient storage; the content cluster is out of memory or disk space. |
##### HTTP response headers
| Header | Values | Description |
| --- | --- | --- |
| X-Vespa-Ignored-Operation | true |
Will be present and set to the string 'true' if (and only if) the operation was ignored during routing. Operations are ignored when not matching the [document selection](../writing/document-selector-language.html) for _all_ content clusters, e.g., for expired documents. Since Vespa 8.680
|
| X-Vespa-Ignored-Fields | true |
Will be present and set to 'true' only when a put or update contains one or more fields which were [ignored since they are not present in the document type](../applications/services/container.html#ignore-undefined-fields). Such operations will be applied exactly as if they did not contain the field operations referencing non-existing fields. References to non-existing fields in field _paths_ are not detected.
|
| Content-Type | `application/json` or `application/jsonl` |
The [media type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/MIME_types) (MIME type) of the response body.
Either `application/json` for [JSON](#json) responses or `application/jsonl` for [JSON Lines](#json-lines) (JSONL) responses.
The content type may include additional parameters such as `charset`.
Example header:
```
Content-Type: application/json; charset=UTF-8
```
|
##### Response formats
Responses are by default in JSON format. [Streaming visiting](#stream) supports an optional [JSON Lines](#json-lines) (JSONL) response format since Vespa 8.593.
###### JSON
JSON responses have the following fields:
| Field | Description |
| --- | --- |
| pathId | Request URL path — always included. |
| message | An error message — included for all failed requests. |
| id | Document ID — always included for single document operations, including _Get_. |
| fields | The requested document fields — included for successful _Get_ operations. |
| documents[] | Array of documents in a visit result — each document has the _id_ and _fields_. |
| documentCount | Number of visited and selected documents. If [includeRemoves](#includeRemoves) is `true`, this also includes the number of returned removes (tombstones). |
| continuation | Token to be used to get the next chunk of the corpus - see [continuation](#continuation). |
GET can include a `fields` object if a document was found in a _GET_ request
```json
{
"pathId": "",
"id": "",
"fields": {
}
}
```
A GET _visit_ result can include an array of `documents` plus a [continuation](#continuation):
```json
{
"pathId": "",
"documents": [
{
"id": "",
"fields": {
}
}
],
"continuation": "