Written by dap
on 

Understanding SMF properties

SMF is the illumos system for managing traditional Unix services (long-lived background processes, usually). It’s quite rich in order to correctly accommodate a lot of different use cases. But it sometimes exposes that complexity to users even when they’re trying to do something simple. For an example, here’s one of my all-time favorite quotes from a manual page, the page for svcprop(1), which prints out SMF properties:

The effective properties of a service are its directly attached properties. The effective properties of a service instance are the union of properties in the composed view of its running snapshot and the properties in nonpersistent property groups in the composed view of the instance’s directly attached properties. See smf(7) for an explanation of property composition.

— svcprop(1) manual page
https://illumos.org/man/1/svcprop

If you can grok what this (and the containing paragraph) are saying, you’ll find that a lot of SMF makes more sense! But it’s awfully hard to figure it out from the text alone. In this post, I’ll walk through an example using a demo service and the svcprop(1) tool to show the details.

This post is a little dry because it walks through so many different cases. But I hope it’ll be a useful tutorial for people trying to understand this material and a reference for when things go wrong.

If you just want a quick reference, check out the Visual summary below.

Prerequisites

I assume you’re at least somewhat familiar with SMF, but here’s a quick summary. See the smf(7) and svcprop(1) manual pages for details.

Service: A service is a sort of template that defines common configuration and behavior for one or more instances (e.g., site/demo/service1). These instances are usually long-running processes — i.e., traditional Unix services.

Instance: An instance is a running copy of a service (e.g., site/demo/service1:instance1), with its own state and potentially customized configuration.

Very often, a service has only one instance called default and the distinction between service (e.g., svc:/network/nfs/server) and instance (svc:/network/nfs/server:default) is not that useful. But it can be useful when you have a few instances that differ only slightly. For example, the console-login service has different instances for the different virtual terminals that it can run on:

$ svcs console-login
STATE          STIME    FMRI
disabled       Nov_08   svc:/system/console-login:vt2
disabled       Nov_08   svc:/system/console-login:vt3
disabled       Nov_08   svc:/system/console-login:vt4
disabled       Nov_08   svc:/system/console-login:vt5
disabled       Nov_08   svc:/system/console-login:vt6
online         Nov_08   svc:/system/console-login:default

These services have only slightly different configuration.

Property groups and properties: Properties are name-value pairs that serve as either configuration or status for a service or instance. Property groups organize related properties. Property groups (and their properties) can be attached either to services or instances.

Property composition: Depending on how you ask for a property on an instance, SMF may apply inheritance rules: if the property exists on the instance, use that value; otherwise, fall back to the service’s value (if it exists).

Snapshots: Snapshots are read-only views of an instance’s properties at specific points in its lifecycle. The most important ones are initial (captured when the instance is created), start (captured when the instance successfully transitions to online), and running (the current effective configuration).

Among other things, snapshots solve the problem of an administrator wanting to change a bunch of properties together and apply them all at once. You wouldn’t want an instance that restarts at the wrong time to pick up a half-updated set of properties. So when you change properties, those changes are reflected in the directly-attached properties, but not the running snapshot of an instance. The administrator refreshes the instance to propagate changes to the running snapshot, causing them to take effect.

Our example service

Let’s work through a concrete example to see how all of this fits together. We’ll create a service with some property groups, import it, make changes, and observe how the different views behave.

Here’s the manifest we’ll use:

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type='manifest' name='demo:service1'>

<service name='site/demo/service1' type='service' version='1'>

	<!-- Make this a transient service so that we don't need any meaningful methods. -->
	<property_group name='startd' type='framework'>
		<propval name='duration' type='astring' value='transient'/>
	</property_group>

	<!-- Property group pg1 on the service -->
	<property_group name='pg1' type='application'>
		<propval name='prop1' type='astring' value='value1_service'/>
		<propval name='prop2' type='astring' value='value2_service'/>
	</property_group>

	<!-- Property group pg2 on the service -->
	<property_group name='pg2' type='application'>
		<propval name='prop3' type='astring' value='value3_service'/>
	</property_group>

	<instance name='instance1' enabled='false'>

		<!-- These start/stop methods do nothing -->
		<exec_method type='method' name='start' exec=':true' timeout_seconds='60' />
		<exec_method type='method' name='stop' exec=':true' timeout_seconds='60' />

		<!-- Property group pg1 on the instance (overrides prop1) -->
		<property_group name='pg1' type='application'>
			<propval name='prop1' type='astring' value='value1_instance'/>
		</property_group>

		<!-- Property group pg3 on the instance -->
		<property_group name='pg3' type='application'>
			<propval name='prop4' type='astring' value='value4_instance'/>
		</property_group>
	</instance>

	<stability value='Unstable' />

	<template>
		<common_name>
			<loctext xml:lang='C'>SMF Property Demo Service</loctext>
		</common_name>
	</template>
</service>
</service_bundle>

Our service has two property groups:

  • pg1 (with prop1 and prop2), and

  • pg2 (with prop3).

The instance defines property groups:

  • pg1 (same as on the service) with just prop1.

  • pg3 (with prop4)

As we’ll see, pg1/prop1 gets overridden, while pg1/prop2 does not.

Examining properties

First, we import the manifest:

$ svccfg import demo_manifest.xml

Now let’s look at the properties. We’ll use three different svcprop invocations:

  1. svcprop -C shows directly-attached properties (no composition, just what’s directly associated with that service or instance)

  2. svcprop -c shows the composed view (uses inheritance)

  3. svcprop (no flags) shows the effective properties (the "current" configuration)

Here’s what we see for the service’s directly-attached properties (svcprop -C):

$ svcprop -p pg1 -p pg2 -C svc:/site/demo/service1 | sort
pg1/prop1 astring value1_service
pg1/prop2 astring value2_service
pg2/prop3 astring value3_service

Easy enough: we see exactly what was in the manifest at the service level. I used the -p flag to select the property groups we’re interested in just to ignore some of the other stuff that’s there by default.

Now for the instance’s directly-attached properties. Here I’ll ask for pg3, too:

$ svcprop -p pg1 -p pg2 -p pg3 -C svc:/site/demo/service1:instance1 | sort
svcprop: Couldn't find property group `pg2' for instance `svc:/site/demo/service1:instance1'.
pg1/prop1 astring value1_instance
pg3/prop4 astring value4_instance

The values we see here are the ones defined in the manifest at the instance level. Notice that pg2 is not there (we get an error) because it’s not directly attached to the instance at all — it only exists in the service.

Now let’s look at the composed view of the instance and service properties (svcprop -c):

$ svcprop -p pg1 -p pg2 -p pg3 -c svc:/site/demo/service1:instance1 | sort
pg1/prop1 astring value1_instance
pg1/prop2 astring value2_service
pg2/prop3 astring value3_service
pg3/prop4 astring value4_instance

It’s starting to get a little more interesting:

  • We see all the properties in all of the property groups defined at both the service-level and the instance-level.

  • pg1/prop1: the value comes from the instance because this value is directly attached to the instance and that always overrides service-level properties in the composed view

  • pg1/prop2: the value comes from the service because it’s not defined on the instance and we’re looking at the composed view

  • pg2/prop3: the value comes from the service because it’s not defined on the instance and we’re looking at the composed view

  • pg3/prop4: the value comes from the instance because it’s only defined at all at the instance level

Finally, we get to the effective properties of the instance:

$ svcprop -p pg1 -p pg2 -p pg3 svc:/site/demo/service1:instance1 | sort
pg1/prop1 astring value1_instance
pg1/prop2 astring value2_service
pg2/prop3 astring value3_service
pg3/prop4 astring value4_instance

This is exactly the same — for the same reasons. We’ll see how this can differ in just a minute.

Changing properties without refreshing

Let’s change pg1/prop2 on the service:

$ svccfg -s site/demo/service1 setprop pg1/prop2 = astring: "value2_service_v2"

Note that we haven’t refreshed anything yet. What do we see now at the service level?

$ svcprop -p pg1 -C svc:/site/demo/service1 | sort
pg1/prop1 astring value1_service
pg1/prop2 astring value2_service_v2

The new value is immediately reflected when we ask for the service’s directly-attached properties.

What about the instance? Its directly-attached properties are unchanged because they never contained this property anyway:

$ svcprop -p pg1 -C svc:/site/demo/service1:instance1
pg1/prop1 astring value1_instance

What about the composed view?

$ svcprop -p pg1 -c svc:/site/demo/service1:instance1 | sort
pg1/prop1 astring value1_instance
pg1/prop2 astring value2_service_v2

The composed view of the instance’s properties reflects the change. So has the service-level change propagated to the instance? Actually, no. Check out the effective properties (default svcprop behavior):

$ svcprop -p pg1 svc:/site/demo/service1:instance1 | sort
pg1/prop1 astring value1_instance
pg1/prop2 astring value2_service

As mentioned above, that’s because property changes don’t get applied to instances until the instance is refreshed.

Note that this behavior has nothing to do with the fact that we changed a service-level property. We can see the same behavior if we change an instance-level property. First, set it:

$ svccfg -s site/demo/service1:instance1 setprop pg3/prop4 = astring: "value4_instance_v2"

See the change reflected in the directly-attached instance properties and composed view:

$ svcprop -p pg3 -C svc:/site/demo/service1:instance1
pg3/prop4 astring value4_instance_v2

$ svcprop -p pg3 -c svc:/site/demo/service1:instance1
pg3/prop4 astring value4_instance_v2

But it’s not reflected in the effective properties yet:

$ svcprop -p pg3 svc:/site/demo/service1:instance1
pg3/prop4 astring value4_instance

We can be really explicit by looking at specific snapshots. The initial snapshot shows the values when the service and instance were created:

$ svcprop -s initial -p pg1 -p pg2 -p pg3 svc:/site/demo/service1:instance1 | sort
pg1/prop1 astring value1_instance
pg1/prop2 astring value2_service
pg2/prop3 astring value3_service
pg3/prop4 astring value4_instance

The running snapshot shows the same:

$ svcprop -s running -p pg1 -p pg2 -p pg3 svc:/site/demo/service1:instance1 | sort
pg1/prop1 astring value1_instance
pg1/prop2 astring value2_service
pg2/prop3 astring value3_service
pg3/prop4 astring value4_instance

Refreshing the instance

Now let’s refresh the instance:

$ svcadm refresh site/demo/service1:instance1

Now, the effective properties reflect the changes to both the service-level and instance-level properties:

$ svcprop -p pg1 -p pg2 -p pg3 svc:/site/demo/service1:instance1 | sort
pg1/prop1 astring value1_instance
pg1/prop2 astring value2_service_v2
pg2/prop3 astring value3_service
pg3/prop4 astring value4_instance_v2

We can see that the running snapshot now reflects the changes:

$ svcprop -s running -p pg1 -p pg2 -p pg3 svc:/site/demo/service1:instance1 | sort
pg1/prop1 astring value1_instance
pg1/prop2 astring value2_service_v2
pg2/prop3 astring value3_service
pg3/prop4 astring value4_instance_v2

start vs. running snapshots

Up to this point, our instance wasn’t even running. Let’s start it and change a property so that we can see the difference between the running and start snapshots.

First, enable the instance:

$ svcadm enable -s svc:/site/demo/service1:instance1

Now let’s change a property:

$ svccfg -s site/demo/service1:instance1 setprop pg3/prop4 = astring: "value4_instance_v3"

As you might expect from the example above, this is not reflected in the instance’s effective properties yet:

$ svcprop -p pg3 svc:/site/demo/service1:instance1
pg3/prop4 astring value4_instance_v2

It’s not in the running snapshot yet:

$ svcprop -s running -p pg3 svc:/site/demo/service1:instance1
pg3/prop4 astring value4_instance_v2

In fact, even if we restart the instance at this point, it still won’t be reflected. This tripped me up as a new SMF user:

$ svcadm restart svc:/site/demo/service1:instance1
$ svcprop -p pg3 svc:/site/demo/service1:instance1
pg3/prop4 astring value4_instance_v2

We have to refresh it to see the change:

$ svcadm refresh site/demo/service1:instance1
$ svcprop -p pg3 svc:/site/demo/service1:instance1
pg3/prop4 astring value4_instance_v3

Once we do, we find that it’s part of the running snapshot:

$ svcprop -s running -p pg3 svc:/site/demo/service1:instance1
pg3/prop4 astring value4_instance_v3

But it’s still not part of the start snapshot!

$ svcprop -s start -p pg3 svc:/site/demo/service1:instance1
pg3/prop4 astring value4_instance_v2

The start snapshot captured the composed configuration at the moment the instance started. That was before we changed the property. We refreshed the instance, which propagates property changes to the running snapshot, but we didn’t actually restart the service. If we do that:

$ svcadm restart svc:/site/demo/service1:instance1

then the new start snapshot will reflect the change:

$ svcprop -s start -p pg3 svc:/site/demo/service1:instance1
pg3/prop4 astring value4_instance_v3
Note
Refresh vs. restart

In SMF, it’s important to understand the difference between restart and refresh.

Refreshing an instance does two things:

  1. Applies any property changes that have been made since the last refresh. More precisely: refreshing propagates changes made to the service and instance properties to the current running snapshot, which also propagates them to the set of effective properties.

  2. Invokes an instance’s refresh method, if one has been defined. That is: you can define a command to be invoked when an instance is refreshed. Traditional services would use :kill -HUP, which sends SIGHUP to the service, which causes it to reread its configuration (which, in this case, would come from SMF).

You can also restart an instance, which just invokes the stop and start methods. It doesn’t refresh the instance. So the instance won’t pick up pending SMF property changes. (Of course, if the service reads from a config file, it will pick those changes up.)

There are three common cases:

  1. If the instance stores configuration in SMF and has a refresh method, then svcadm refresh is enough to apply pending property changes (both in terms of SMF and in terms of propagating them to the running process).

  2. If the instance stores configuration in SMF but does not define a refresh method, then you have to svcadm refresh it to apply property changes and then svcadm restart it to have the program pick those up.

  3. If the instance stores configuration elsewhere (e.g., a configuration file), you usually ignore SMF properties altogether and would just svcadm restart it to have it load its configuration again.

Visual summary

Different views of SMF properties

Conclusion

As a human administrator, you often want to just set some property and restart a service to have it take effect. With SMF, it’s often not that simple. That’s because it’s trying to be a robust platform for both administrators and automation. As an example, it’s trying to handle cases where multiple properties may be changed in succession but those changes should propagate atomically to the service. This complexity also leads to the various different views related to "the properties for an instance". This can be annoying for new users, but here are the key points to remember:

  1. Properties can be defined at either the service level or instance level.

  2. Composition happens at the instance level: Instance properties override service properties; otherwise, instances inherit from the service.

  3. Running instances are isolated from configuration changes: The effective properties come from the running snapshot, which only updates when the instance is refreshed. This is a feature, not a bug: it prevents live instances from seeing configuration changes mid-flight.

  4. svcprop provides three distinct views:

    • svcprop shows effective properties and is usually what you want

    • svcprop -C shows directly-attached properties (useful for debugging to figure out where properties are defined)

    • svcprop -c shows the composed view (what would the instance get if it restarted now?)