HashiConf 2024 Now streaming live from Boston! Attend for free
Nomad
Nomad Home

service Block

Placementjob -> group -> service
job -> group -> task -> service

The service block instructs Nomad to register a service with the specified provider; Nomad or Consul. This section of the documentation will discuss the configuration, but please also read the Nomad service discovery documentation for more detailed information about the external integrations.

job "docs" {
  group "example" {
    task "server" {
      service {
        tags = ["leader", "mysql"]

        port = "db"

        provider = "consul"

        weights {
          passing = 5
          warning = 1
        }

        meta {
          meta = "for your service"
        }

        check {
          type     = "tcp"
          port     = "db"
          interval = "10s"
          timeout  = "2s"
        }

        check {
          type     = "http"
          name     = "app_health"
          path     = "/health"
          interval = "20s"
          timeout  = "5s"

          check_restart {
            limit = 3
            grace = "90s"
            ignore_warnings = false
          }
        }

        identity {
          aud = ["consul.io"]
        }
      }
    }
  }
}

This section of the documentation only cover the job file fields and blocks for service discovery. For more details on using Nomad with Consul please see the Consul integration documentation.

The service block can also be at task group level. This enables services in the same task group to opt into Consul Service Mesh integration.

service Parameters

  • provider (string: "consul") - Specifies the service registration provider to use for service registrations. Valid options are either consul or nomad. All services within a single task group must utilise the same provider value.

  • cluster (string: "default") Enterprise - Specifies the Consul cluster to use, when the provider is "consul". The Nomad client will retrieve a Consul token from the cluster configured in the agent configuration with the same consul.name. In Nomad Community Edition, this field is ignored.

  • check (Check: nil) - Specifies a health check associated with the service. This can be specified multiple times to define multiple checks for the service. At this time, a check using the Nomad provider supports tcp and http checks. The Consul integration supports the grpc, http, script1, and tcp checks.

  • weights (Weights: nil) - Specifies how a service instance is weighted in a DNS SRV request based on the service's health status. Only available where provider = "consul".

  • connect - Configures the Consul Connect integration. Only available on group services and where provider = "consul".

  • identity (Identity: nil) - Specifies a Workload Identity to use when obtaining Service Identity tokens from Consul to register the service. Only available where provider = "consul". Typically this can be omitted so that Nomad will fall back to the server's consul.service_identity block.

  • name (string: "<job>-<taskgroup>-<task>") - Specifies the name this service will be advertised as in Consul. If not supplied, this will default to the name of the job, task group, and task concatenated together with a dash, like "docs-example-server". Each service must have a unique name within the cluster. Names must adhere to RFC-1123 §2.1 and are limited to alphanumeric and hyphen characters (i.e. [a-z0-9\-]), and be less than 64 characters in length.

    In addition to the standard Nomad interpolation, the following keys are also available:

    Validation of the name occurs in two parts. When the job is registered, an initial validation pass checks that the service name adheres to RFC-1123 §2.1 and the length limit, excluding any variables requiring interpolation. Once the client receives the service and all interpretable values are available, the service name will be interpolated and revalidated. This can cause certain service names to pass validation at submit time but fail at runtime.

  • port (string: <optional>) - Specifies the port to advertise for this service. The value of port depends on which address_mode is being used:

    • alloc - Advertise the mapped to value of the labeled port and the allocation address. If a to value is not set, the port falls back to using the allocated host port. The port field may be a numeric port or a port label specified in the same group's network block.

    • driver - Advertise the port determined by the driver (e.g. Docker). The port may be a numeric port or a port label specified in the driver's ports field.

    • host - Advertise the host port for this service. port must match a port label specified in the network block.

  • tags (array<string>: []) - Specifies the list of tags to associate with this service. If this is not supplied, no tags will be assigned to the service when it is registered.

  • canary_tags (array<string>: []) - Specifies the list of tags to associate with this service when the service is part of an allocation that is currently a canary. Once the canary is promoted, the registered tags will be updated to those specified in the tags parameter. If this is not supplied, the registered tags will be equal to that of the tags parameter.

  • enable_tag_override (bool: false) - Enables users of Consul's Catalog API to make changes to the tags of a service without having those changes be overwritten by Consul's anti-entropy mechanism. See Consul documentation for more information. Only available where provider = "consul".

  • address (string: <optional>) - Specifies a custom address to advertise in Consul or Nomad service registration. If set, address_mode must be in auto mode. Useful with interpolation - for example to advertise the public IP address of an AWS EC2 instance set this to ${attr.unique.platform.aws.public-ipv4}.

  • tagged_addresses (map<string|string> - Specifies custom tagged addresses to advertise in the Consul service registration. Only available where provider = "consul".

  • address_mode (string: "auto") - Specifies which address (host, alloc or driver-specific) this service should advertise. See below for examples. Valid options are:

    • alloc - For allocations which create a network namespace, this address mode uses the IP address inside the namespace. Can only be used with "bridge" and "cni" networking modes. A numeric port may be specified for situations where no port mapping is necessary. This mode can only be set for services which are defined in a "group" block.

    • auto - Allows the driver to determine whether the host or driver address should be used. Defaults to host and only implemented by Docker. If you use a Docker network plugin such as weave, Docker will automatically use its address.

    • driver - Use the IP specified by the driver, and the port specified in a port map. A numeric port may be specified since port maps aren't required by all network plugins. Useful for advertising SDN and overlay network addresses. Task will fail if driver network cannot be determined. Only implemented for Docker. This mode can only be set for services which are defined in a "task" block.

    • host - Use the host IP and port.

  • task (string: "") - Specifies the name of the Nomad task associated with this service definition. Only available on group services. Must be set if this service definition represents a Consul Connect-native service and there is more than one task in the task group.

  • meta ([Meta][]: nil) - Specifies a key-value map that annotates the Consul service with user-defined metadata. Only available where provider = "consul".

  • canary_meta ([Meta][]: nil) - Specifies a key-value map that annotates the Consul service with user-defined metadata when the service is part of an allocation that is currently a canary. Once the canary is promoted, the registered meta will be updated to those specified in the meta parameter. If this is not supplied, the registered meta will be set to that of the meta parameter. Only available where provider = "consul".

  • on_update (string: "require_healthy") - Specifies how checks should be evaluated when determining deployment health (including a job's initial deployment). This allows job submitters to define certain checks as readiness checks, progressing a deployment even if the Service's checks are not yet healthy. Checks inherit the Service's value by default. The check status is not altered in Consul and is only used to determine the check's health during an update.

    • require_healthy - In order for Nomad to consider the check healthy during an update it must report as healthy.

    • ignore_warnings - If a Service Check reports as warning, Nomad will treat the check as healthy. The Check will still be in a warning state in Consul.

    • ignore - Any status will be treated as healthy.

    Caveat: on_update is only compatible with certain check_restart configurations. on_update = "ignore_warnings" requires that check_restart.ignore_warnings = true. check_restart can however specify ignore_warnings = true with on_update = "require_healthy". If on_update is set to ignore, check_restart must be omitted entirely.

service Lifecycle

Nomad manages registering, updating, and deregistering services with the service provider. It is important to understand when each of these steps happens and how they can be customized.

Registration: Nomad will register group services and checks before starting any tasks. Services and checks for a specific task are registered after the task has started.

Updating: If a service or check definition is updated, Nomad will update the service in the provider as well. This update happens without restarting a task.

Deregistering: If a running task with a service block exits, the services and checks are immediately deregistered from the provider without delay. If, however, Nomad needs to kill a running task, the task is killed in the following order:

  1. Immediately remove the services and checks from the provider. This stops new traffic from being routed to the task that is being killed.
  2. If shutdown_delay is set, wait the configured duration before proceeding to step 3. Setting a shutdown_delay can be useful if the application itself doesn't handle graceful shutdowns based on the kill_signal. The configured delay will provide a period of time in which the service is no longer registered in the provider, and thus is not receiving additional requests, but hasn't been signalled to shutdown. This allows the application time to complete the requests and become idle.
  3. Send the kill_signal to the task and wait for the task to exit. The task should use this time to gracefully drain and finish any existing requests.
  4. If the task has not exited after the kill_timeout, Nomad will force kill the application.

service Examples

The following examples only show the service blocks. Remember that the service block is only valid in the placements listed above.

Basic Service

This example registers a service named "load-balancer" with no health checks using the Nomad provider:

service {
  name     = "load-balancer"
  port     = "lb"
  provider = "nomad"
}

This example registers a service named "load-balancer" with no health checks using the Consul provider:

service {
  name = "load-balancer"
  port = "lb"
}

These examples must be accompanied by a network block which defines a static or dynamic port labeled "lb". For example:

network {
  port "lb" {}
}

Using Driver Address Mode

The Docker driver supports the driver setting for the address_mode parameter in both service and check blocks. The driver address mode allows advertising and health checking the IP and port assigned to a task by the driver. This way, if you're using a network plugin like Weave with Docker, you can advertise the Weave address in Consul instead of the host's address.

For example if you were running the example Redis job in an environment with Weave but Consul was running on the host you could use the following configuration:

job "example" {
  datacenters = ["dc1"]

  group "cache" {
    network {
      port "db" {
        to = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis:7"
        network_mode = "weave"
        ports = ["db"]
      }

      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
      }

      service {
        name = "weave-redis"
        port = "db"
        check {
          name     = "host-redis-check"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

No explicit address_mode required.

Services default to the auto address mode. When a Docker network mode other than "host" or "bridge" is used, services will automatically advertise the driver's address (in this case Weave's). The service will advertise the container's port: 6379.

However since Consul is often run on the host without access to the Weave network, check blocks default to host address mode. The TCP check will run against the host's IP and the dynamic host port assigned by Nomad.

Note that the check still inherits the service block's db port label, but each will resolve the port label according to their address mode.

If Consul has access to the Weave network the job could be configured like this:

job "example" {
  datacenters = ["dc1"]
  group "cache" {

    task "redis" {
      driver = "docker"

      config {
        image = "redis:7"
        network_mode = "weave"
        # No port map required.
      }

      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
      }

      service {
        name = "weave-redis"
        port = 6379
        address_mode = "driver"
        check {
          name     = "host-redis-check"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
          port     = 6379

          address_mode = "driver"
        }
      }
    }
  }
}

In this case Nomad doesn't need to assign Redis any host ports. The service and check blocks can both specify the port number to advertise and check directly since Nomad isn't managing any port assignments.

IPv6 Docker containers

The Docker driver supports the advertise_ipv6_address parameter in its configuration.

Services will automatically advertise the IPv6 address when advertise_ipv6_address is used.

Unlike services, checks do not have an auto address mode as there's no way for Nomad to know which is the best address to use for checks. Consul needs access to the address for any HTTP or TCP checks.

So you have to set address_mode parameter in the check block to driver.

For example using auto address mode:

job "example" {
  datacenters = ["dc1"]
  group "cache" {

    network {
      port "db" {
        to = 6379
      }
    }


    task "redis" {
      driver = "docker"

      config {
        image = "redis:7"
        advertise_ipv6_address = true
        ports = ["db"]
      }

      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
      }

      service {
        name = "ipv6-redis"
        port = "db"
        check {
          name     = "ipv6-redis-check"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
          port     = "db"
          address_mode = "driver"
        }
      }
    }
  }
}

Or using address_mode=driver for service and check with numeric ports:

job "example" {
  datacenters = ["dc1"]

  group "cache" {

    task "redis" {
      driver = "docker"

      config {
        image = "redis:7"
        advertise_ipv6_address = true
        # No port map required.
      }

      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
      }

      service {
        name = "ipv6-redis"
        port = 6379
        address_mode = "driver"
        check {
          name     = "ipv6-redis-check"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
          port     = 6379
          address_mode = "driver"
        }
      }
    }
  }
}

The service and check blocks can both specify the port number to advertise and check directly since Nomad isn't managing any port assignments.

Edit this page on GitHub