In the past, we have discussed various ways on how to debug and troubleshoot potential issues during snap development. The ability to quickly iterate, resolve build process hurdles and publish the application in a timely manner is essential to a robust, positive development experience. Today, we would like to outline a few basic tips and tricks focused around snaps that contain services.

Inspect service status & manually run services

With snaps that contain services, your most likely point of friction will be after the build process. To that end, you should install the built snap locally and then check that it runs correctly, and that the services start successfully.

snap install “snapname” --dangerous

The –dangerous flag is required because your locally built snap does not originate from the Snap Store, and it has not been cryptographically signed.

Once the snap is installed, you can check whether the services contained therein run. Let’s demonstrate with Nextcloud, which is a fairly complex snap that contains multiple applications, including various services:

snap services nextcloud

Service                    Startup  Current  Notes
nextcloud.apache           enabled  active   -
nextcloud.mdns-publisher   enabled  active   -
nextcloud.mysql            enabled  active   -
nextcloud.nextcloud-cron   enabled  active   -
nextcloud.nextcloud-fixer  enabled  active   -
nextcloud.php-fpm          enabled  active   -
nextcloud.redis-server     enabled  active   -
nextcloud.renew-certs      enabled  active   -

In this case, we can see that the services are enabled and active. However, in case of a failure, you would most likely see something like the Current column reads “inactive”.

Service                    Startup  Current  Notes
starwars.tng               enabled  inactive   -

At this point, you can attempt starting the failed service manually, to see whether there are any useful messages (standard output and standard error) printed in your terminal.

sudo snap start “snap name”
sudo snap snart “snap name.”service name”

Alternatively, you can “run” the service, like you would any application – with all the expected parameters that the service requires to see its behavior. You should also see messages and errors printed inside your terminal windows. For instance:

sudo snap run nextcloud.mdns-publisher
2020/05/27 12:20:45 Publishing virt.local -> [10.0.2.15 fe80::c75f:601a:88b1:3dab] with 60-second TTL

Inspect logs

The first step in trying to understand the potential service startup failure is to examine the logs. In general, this is always a useful course of action – including applications that start successfully. While it is possible that the services run, they may be misconfigured or may require tweaking. The command syntax is as follows:

snap logs “service snap name”

Again, let’s examine the Nextcloud example. 

snap logs nextcloud
2020-05-26T12:11:46Z nextcloud.apache[3229]: done
2020-05-26T12:11:46Z nextcloud.apache[3229]: Configuring nextcloud…
2020-05-26T12:11:47Z nextcloud.apache[3136]: All set! Running httpd…
2020-05-26T12:11:47Z nextcloud.apache[4196]: No certificates are active: using HTTP only
2020-05-26T12:11:47Z nextcloud.nextcloud-cron[3094]: Waiting for Nextcloud config dir... done
2020-05-26T12:11:47Z nextcloud.apache[4229]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
2020-05-26T12:11:48Z nextcloud.nextcloud-fixer[3022]: done
2020-05-26T12:11:48Z nextcloud.nextcloud-fixer[3022]: Waiting for Nextcloud to be installed...

Here, from the end of the log, we can see that the nextcloud.apache service is running, however it could not determine the server’s FQDN. In some scenarios, this is perfectly normal, and you can ignore the message. In others, this would be a useful piece of information that would require changing the server configuration.

You can fine tune the level of verbosity and information displayed. For a snap with multiple services, you may only be interested in a specific service:

snap logs “snap name.service name”

Below, we are only looking for log information on the Nextcloud MySQL service. The optional -n flag determines how many lines of log data from the end of the file should be displayed. You may also use the -f flag to see to follow the log, so that any new messages will be appended and printed to the standard output.

snap logs -n=all nextcloud.mysql
2020-05-26T12:11:41Z nextcloud.mysql[3767]: mysql.time_zone_transition_type                               OK
2020-05-26T12:11:41Z nextcloud.mysql[3767]: mysql.user        OK
2020-05-26T12:11:41Z nextcloud.mysql[3767]: The sys schema is already up to date (version 1.5.2).
2020-05-26T12:11:41Z nextcloud.mysql[3767]: Checking databases.
2020-05-26T12:11:41Z nextcloud.mysql[3767]: sys.sys_config    OK
2020-05-26T12:11:41Z nextcloud.mysql[3767]: Upgrade process completed successfully.
2020-05-26T12:11:41Z nextcloud.mysql[3767]: Checking if update is needed.
2020-05-26T12:11:41Z nextcloud.mysql[3050]: Restarting mysql server after upgrade…
2020-05-26T12:11:41Z nextcloud.mysql[3828]: Shutting down MySQL
2020-05-26T12:11:43Z nextcloud.mysql[3828]: .. *
2020-05-26T12:11:43Z nextcloud.mysql[3864]: Starting MySQL
2020-05-26T12:11:44Z nextcloud.mysql[3864]: . *

Optionally, you can also use the system journalctl command to see the snap logs:

journalctl -ef -u snap.”snap name”.”service name”

This command will show the log for the unit file that corresponds to the written name (-u), will show the end of the log (-e), and will follow it (-f), so you will see any new messages also being printed to the standard output, which can be quite useful when troubleshooting services in real time.

The logs are quite useful in that you may not necessarily see an exact reason for a service startup failure on the command line, even if you manually run the snap start command. A service may be configured incorrectly, the messages could be saved to a separate log file.

snap services igor
Service                  Startup  Current   Notes
igor.main                enabled  inactive  -
igor.secondary           enabled  inactive  -

sudo snap start igor.secondary
Started.

However, while the message reads Started, the service is still in the inactive state. We now inspect the logs, or manually run the igor.secondary service to get additional information, and hopefully, be able to resolve the problem.

2020-05-28T12:12:56Z igor.secondary[18440]: + tee -a /root/snap/igor/x3/logs/secondary.log
2020-05-28T12:12:56Z igor.secondary[18440]: + /snap/igor/x3/usr/share/secondary/startup.sh --host=localhost --domain=locutus --port=7747 --apis=rest,xmpp
2020-05-28T12:12:56Z igor.secondary[18440]: Error: Could not find or load main class SECONDARY_LOGFILE:
2020-05-28T12:12:56Z igor.secondary[18440]: Caused by: java.lang.ClassNotFoundException: SECONDARY_LOGFILE:
2020-05-28T12:12:56Z systemd[1]: snap.igor.secondary.service: Succeeded.

Debug failed run

As we’ve outlined in the past snap troubleshooting tutorials, you can step into the snap environment by using the –shell flag. This allows you to inspect the snap structure, as well as verify the value of any environment variables.

snap run --shell “snap name”
snap run --shell “snap name”.”service name”

You can now try to manually check and configure the application environment, and then run any command in the service chain to see where the failure occurs, and if there are any misconfigurations that need to be addressed.

Install hook

Some snaps may include services that cannot run immediately after the installation, as they require a manual post-install configuration, and therefore will fail to start correctly. You can change the default behavior by using an install hook.

Hooks are executable files that run within a snap’s confined environment when a certain action occurs. If you require specific commands to be triggered in different phases of a snap lifecycle, you can create hooks inside the snap’s hooks folder – “project workspace”/snap/hooks, and they will then be parsed and run by the snapd service at the required phase in the lifecycle. For instance, the install hook will run only once on first installation, the configure hook will run on installation and any subsequent refresh, and so on.

Hooks also obey snap confinement rules, so if they require access to resources outside the snap, you will need to provide granular overrides using interfaces, like access to the network, the ability to bind ports, or access remote storage.

hooks:
  configure:
    plugs: [network, network-bind, removable-media]
  pre-refresh:
    plugs: [network, network-bind, removable-media]

If your service depends on a specific configuration that is not available when the snap is installed, you can prevent the startup, and then manually change the service state. This can be done by using –enable and –disable flags when starting or stopping a service, respectively.

snap start --enable “snap name”.”service name”
snap stop --disable “snap name”.”service name”

For instance:

snap stop --disable nextcloud.mdns-publisher

snap services nextcloud
Service                    Startup   Current   Notes
nextcloud.apache           enabled   active    -
nextcloud.mdns-publisher   disabled  inactive  -
nextcloud.mysql            enabled   active    -
nextcloud.nextcloud-cron   enabled   active    -
nextcloud.nextcloud-fixer  enabled   active    -
nextcloud.php-fpm          enabled   active    -
nextcloud.redis-server     enabled   active    -
nextcloud.renew-certs      enabled   active    -

Check ports

Additionally, you can also use the system commands like netstat or ss to check which processes are running, and if they are listening to any ports. This can help you spot any possible misconfigurations, even if the services are seemingly active. For instance, a different process may bind a port that is necessary for the functionality of one of the services in your snap. Or if you are using the parallel installs feature, you may need to make adjustments to your project to allow multiple instances of your services to run without conflicts.

ss -tulpan
Netid    State     Recv-Q    Send-Q          Local Address:Port         Peer Address:Port    Process    
udp      UNCONN    0         0               127.0.0.53%lo:53                0.0.0.0:*                  
udp      ESTAB     0         0            10.0.2.15%enp0s3:68               10.0.2.2:67           
udp      UNCONN    0         0                     0.0.0.0:41523             0.0.0.0:*                  
udp      UNCONN    0         0                     0.0.0.0:631               0.0.0.0:*                  
udp      UNCONN    0         0                     0.0.0.0:5353              0.0.0.0:*                  
udp      UNCONN    0         0                     0.0.0.0:5353              0.0.0.0:*                  
udp      UNCONN    0         0                        [::]:48228                [::]:*                  
udp      UNCONN    0         0                        [::]:5353                 [::]:*                  
udp      UNCONN    0         0                        [::]:5353                 [::]:*                  
tcp      LISTEN    0         4096            127.0.0.53%lo:53                0.0.0.0:*                  
tcp      LISTEN    0         128                   0.0.0.0:22                0.0.0.0:*                  
tcp      LISTEN    0         5                   127.0.0.1:631               0.0.0.0:*                  
tcp      LISTEN    0         511                         *:80                      *:*                  
tcp      LISTEN    0         128                      [::]:22                   [::]:*                  
tcp      LISTEN    0         5                       [::1]:631                  [::]:* 

Summary

At first glance, bundling applications with services as snaps may seem like a difficult task. While the concept of confinement does present a paradigm shift in development, the handling of the network stack remains unchanged. Snaps share the process namespace with the underlying system, and the core ideas of network port binding and associated functionality are identical to the classic Linux development.

In this article, we outlined a number of tools you can use to inspect your services, verify that they are working correctly – and if they are not, dig into the log messages and try to understand what type of issues may exist. If you have any questions or comments on this topic, please join our forum for a discussion.

Photo by todd kent on Unsplash.