Building a (real) Linux daemon with Delphi by Paolo Rossi

[emailx45]

Building a (real) Linux daemon with Delphi
Paolo Rossi - July 11, 2017

Спойлер: Content - 1st part

[SHOWTOGROUPS=4,20]

Building a (real) Linux daemon with Delphi Series

1. Part 1: Process management in Linux
2. Part 2: Build a Linux daemon with Delphi

This will be a multipart article, in this first part I explore the options of launching a (console) Linux program in background, in the next part I will focus on building a proper Linux Daemon with Delphi.

So let's start!

Introduction
In the past weeks I've been busy converting some REST services, built with the Для просмотра ссылки Войди или Зарегистрируйся, to Linux. Compiling the projects to Linux was the easy part, thanks to the great job Embarcadero did on the Linux compiler and the RTL.

For debugging and testing purposes, a console Linux application was sufficient but to deploy the service (actually on a AWS Linux instance) a simple console was not an option, so I've decided to build a Linux Daemon for that purpose.

Problem is that Delphi (at least in 10.2 Tokyo) doesn't support the creation of a Linux Daemon with an application template, so you have to build one for yourself using directly Linux system calls.

Для просмотра ссылки Войди или ЗарегистрируйсяДля просмотра ссылки Войди или Зарегистрируйся
My setup: tools & applications
In order to be able to run, control and run application under Linux you must obviously have some (additional) software installed in your machine, my setup:

• Windows 10 as host OS
• VirtualBox as virtualization software (Для просмотра ссылки Войди или Зарегистрируйся)
• Debian 9 Stretch as Linux OS (Для просмотра ссылки Войди или Зарегистрируйся) or Ubuntu Server 16.04.2 LTS (Для просмотра ссылки Войди или Зарегистрируйся)
• Putty as terminal emulator (Для просмотра ссылки Войди или Зарегистрируйся) or the Ubuntu Bash Shell if you have Windows 10 Anniversary Edition (Для просмотра ссылки Войди или Зарегистрируйся)
• WinSCP as SCP client for Windows (Для просмотра ссылки Войди или Зарегистрируйся) or the Ubuntu Bash Shell if you have Windows 10 Anniversary Edition (Для просмотра ссылки Войди или Зарегистрируйся)
• Delphi 10.2 Tokyo to create applications for Linux, what else? (Для просмотра ссылки Войди или Зарегистрируйся)

Even if you are not an expert on Linux I encourage you to try to install it on a virtual machine, you'll find that the process is very easy and fast.
There are already some articles that explain in detail the process, and I'm actually writing an article on installing Debian 9 from scratch and configuring it for Delphi development, so.. stay tuned.

Launching a process in Linux
First of all if you have the Linux machine already configured for Delphi development, you can create a simple console application in Delphi, this one would be fine for our "launching program" exercise:

Спойлер: Code

program DemoConsole;

{$APPTYPE CONSOLE}

{$R *.res}

uses
System.SysUtils;

begin
try
Writeln('Hello World');
except
on E: Exception do
Writeln(E.ClassName, ': ', E.Message);
end;
end.

Now if you want to be able to launch your program under Linux independently (not via Delphi IDE) you must deploy your application (the "Deploy DemoConsole" menu command in the Delphi IDE). Now you have to launch your application in Linux, how to do it?

Remember that you have already opened several programs by typing ls, apt-get, wget, etc... so, to launch a Linux command (program) you simply have to write the command name in the shell, so let's try with our newly created Delphi program DemoConsole:
paolor@debian-9:˜$ DemoConsole

and...... doesn't work! :-(

Introducing the ./ (dot-slash)
The previous command failed because in Linux (Unix) the current directory is not in the current path (for security reasons) so you have to specify yourself the path:

paolor@debian-9:˜$ ./DemoConsole

where ./ is the current directory

Great, now you program is (finally) running, but you notice that you can't use that console (for other commands and so on) until your process is terminated.

And yes, of course you can always use another console to log into the system

but this is far less than ideal. This is what happens when you start the process with ./DemoConsole:

1. The process DemoConsole is created
2. The process inherits stdin, stdout, and stderr from the shell
3. The process receives SIGHUP signals (possibly) sent to the shell
4. The shell is blocked until the process terminates

As you can see the 3rd and 4th points are bad things and the 2nd is not very useful if it's not an interactive console application (like services)

The & option (starting a background job)
Another option is to launch your process appending a & after the process name:

paolor@debian-9:˜$ ./DemoConsole &

so the console will be immediately available to you! Great! it seems to be the perfect way to launch your service but.. it's not! So here what happens:

1. The process DemoConsole is created.
2. The process inherits stdout/stderr from the shell (not the stdin)
3. As soon as the process tries to read from stdin, the process halts!
4. The process receives SIGHUP signals sent to the shell.

The 3rd item is quite dangerous and killing the shell still kills your process.

nohup and disown on the rescue
So even with & the background process in not detached from the console, to do so we have 2 options: the commands disown and nohup. Basically they do the same thing, they detach the process from the shell isolating it from SIGHUP signals, but they are slightly different though:

to use nohup:

paolor@debian-9:˜$ nohup ./ConsoleDemo &

to use disown:

paolor@debian-9:˜$ ./ConsoleDemo &
paolor@debian-9:˜$ disown

or if you forget to put your program in background:

paolor@debian-9:˜$ ./ConsoleDemo
paolor@debian-9:˜$ <Ctrl-Z>
paolor@debian-9:˜$ bg
paolor@debian-9:˜$ disown

other differences:

• disown doesn't redirect stdout/stderr so you have no way to see messages possibly sent to the (original) shell
• nohup redirect stdout/stderr to nohup.out file, so you can type tail nohup.out to see your messages
• nohup is defined by POSIX while disown is not, this means that some shell have it, some have not.

I largely prefer nohup to disown because it's easier to script, and the nohup.out file is can be useful (although it doesn't replace a proper logging system)

Для просмотра ссылки Войди или ЗарегистрируйсяДля просмотра ссылки Войди или Зарегистрируйся

Conclusions
In this article we explored the options for launching our (standard) console application (that must behave as a service) in Linux and the preferred way seems to be the nohup command. This solution works but a real Linux daemon is by far the better way to deploy "services" on the Linux platform and even more so is the "official" way in UNIX OSes but to be able to build a daemon in Delphi you have to wait the next article, stay tuned!


[/SHOWTOGROUPS]

 

[emailx45]

Спойлер: Content - 2nd part 1

[SHOWTOGROUPS=4,20]
In the first part I showed how to run a normal console application in the background, now is the time to build a real Linux daemon with Delphi!

First of all I want to thank my friend and colleague Для просмотра ссылки Войди или Зарегистрируйся that first built the daemon .dpr skeleton and showed it at the Для просмотра ссылки Войди или Зарегистрируйся Italian conference in June.

What is a Linux (UNIX) daemon
There are 3 basic types of processes in Linux:

• Interactive run interactively by a user at the command line
• Batch submitted from a queue of processes and not associated with the command line
• Daemon background process that is designed to run autonomously, with little or no user intervention

Daemons are recognized by the system as any processes whose parent process has a pid = 1 (the process with 1 as a pid, always represents the process init).

init is always the first process that is started when a Linux computer is booted up and it remains on the system until the computer is turned off. init adopts any process whose parent process dies without waiting for the child process's status.

So, the common method for creating a daemon involves forking (twice), and making the parent (and grandparent) processes die, while the grandchild process begins performing its normal function.

Running in background means that a daemon must be detached from the terminal and runs continuously in a non-interactive mode.

A daemon cannot read/write from/to the stdin, stdout and stderr and to be sure that it doesn't crash readind or writing on them they must be redirected to /dev/null

So, basically the operations needed to create a daemon are:

1. fork the parent process and let it terminate
2. setsid to create a new session detaching from its controlling terminal
3. signals handling to catch/ignore signals
4. fork again and let it terminate to ensure that you get rid of the session leading process
5. close all open file descriptors inherited from the parent process
6. redirect stdin, stdout, stderr to dev/null to ensure that the daemon doesn't crash
7. chdir to change the working directory of the daemon
8. umask to change the file mode mask according to the needs of the daemon.

Для просмотра ссылки Войди или ЗарегистрируйсяДля просмотра ссылки Войди или Зарегистрируйся
The Delphi code
I know that you are impatient to see the code so... here it is!

Спойлер: Code

program delphid;

{$APPTYPE CONSOLE}

uses 
  System.SysUtils, System.IOUtils,
  Posix.Stdlib, Posix.SysStat, Posix.SysTypes, Posix.Unistd, Posix.Signal, Posix.Fcntl,
  Posix.Syslog in 'Posix.Syslog.pas';

const 
  // Missing from linux/StdlibTypes.inc !!! <stdlib.h>
  EXIT_FAILURE = 1;
  EXIT_SUCCESS = 0;

var 
  pid: pid_t;
  fid: Integer;
  idx: Integer;
  running: Boolean;

procedure HandleSignals(SigNum: Integer); cdecl; 
begin 
  case SigNum of
    SIGTERM:
    begin
      running := False;
    end;
    SIGHUP:
    begin
      syslog(LOG_NOTICE, 'daemon: reloading config');
      // Reload configuration
    end;
  end;
end;

begin 
  openlog(nil, LOG_PID or LOG_NDELAY, LOG_DAEMON);

  if getppid() > 1 then
  begin
    pid := fork();
    if pid < 0 then
      raise Exception.Create('Error forking the process');

    if pid > 0 then
      Halt(EXIT_SUCCESS);

    if setsid() < 0 then
      raise Exception.Create('Impossible to create an independent session');

    signal(SIGCHLD, TSignalHandler(SIG_IGN));
    signal(SIGHUP, HandleSignals);
    signal(SIGTERM, HandleSignals);

    pid := fork();
    if pid < 0 then
      raise Exception.Create('Error forking the process');

    if pid > 0 then
      Halt(EXIT_SUCCESS);

    for idx := sysconf(_SC_OPEN_MAX) downto 0 do
      __close(idx);

    fid := __open('/dev/null', O_RDWR);
    dup(fid);
    dup(fid);

    umask(027);

    chdir('/');
  end; 

  running := True;
  try
    while running do
    begin
      // deamon actual code
      Sleep(1000);
    end;

    ExitCode := EXIT_SUCCESS;
  except
    on E: Exception do
    begin
      syslog(LOG_ERR, 'Error: ' + E.Message);
      ExitCode := EXIT_FAILURE;
    end;
  end;

  syslog(LOG_NOTICE, 'daemon stopped');
  closelog();
end.

This is the (minimal) Delphi code to build a real Linux daemon, quite simple right? But despite the simplicity of this code I invite you to read the article because I think that to build a robust daemon application you have to understand every step of the process, so please make yourself comfortable because this is a lengthy article!

Program, unit uses and consts

program delphid;

{$APPTYPE CONSOLE}

uses 
  System.SysUtils, System.IOUtils,
  Posix.Stdlib, Posix.SysStat, Posix.SysTypes,
  Posix.Unistd, Posix.Signal, Posix.Fcntl,
  Posix.Syslog in 'Posix.Syslog.pas';

const 
  // Missing from linux/StdlibTypes.inc !!! <stdlib.h>
  EXIT_FAILURE = 1;
  EXIT_SUCCESS = 0;

As you can see this is a normal console application File -> New -> Other -> Console Application with a bunch of Posix.* uses. These are the POSIX API units (like the Winapi.* counterpart) and you can expect to find const definitions, (external) function declarations, etc... Here I've added the two const EXIT_FAILURE and EXIT_SUCCESS strangely missing from the StdlibTypes.inc file.


The syslog API

The last unit is the Posix.Syslog.pas that contains calls to the syslogd (or compatible) Linux daemon. The syslog API is missing in 10.2 Tokyo (plain or update 1) so remember that this file is required to log to the syslog daemon but it's not required to build a Linux daemon (although IMO a Linux daemon must log via syslogd).

UNIX-like operating systems have an API for programs to send log messages to the system, if an application is considered critical and its state is the system administrator's responsibility then this application must log via syslogd.

Despite the power and flexibility of syslogd, the syslog API is very simple:

• void openlog(char *ident, int option, int facility)
• void syslog (int priority, char *format, ...)
• void closelog(void )

In the code, as you can see, I've called openlog with no custom ident value (the process name will be used), the option value is LOG_PID or LOG_NDELAY to tell the syslogd to log also the pid and to open the connection immediately and, finally, the facility is set to LOG_DAEMON.

openlog(nil, LOG_PID or LOG_NDELAY, LOG_DAEMON);

If you don't call openlog() explicitly, syslog() will call it for you with no arguments.

The call of syslog() is very simple, you have to pass the priority parameter:

• LOG_EMERG: The system is unusable
• LOG_ALERT: Action must be taken immediately
• LOG_CRIT: Critical condition
• LOG_ERR: Generic error condition
• LOG_WARNING: Warning message
• LOG_NOTICE: Normal but important event
• LOG_INFO: Purely informational message
• LOG_DEBUG: Debugging-level message

syslog(LOG_NOTICE, 'daemon: reloading config');

The call of closelog() doesn't need to be explained:

closelog();

The standard location of syslog log is /var/log/syslog:

The syslogd has other features and some are very interesting from a developer's point of view. I recommend to read some documentation about syslogd and its configuration, in addition I'm thinking to write a small article about the logging features found in Linux distros.

The routine HandleSignals

// 1. If SIGTERM is received, shut down the daemon and exit cleanly.
// 2. If SIGHUP is received, reload the configuration files, if applies.
procedure HandleSignals(SigNum: Integer); cdecl; 
begin 
  case SigNum of
    SIGTERM:
    begin
      running := False;
    end;
    SIGHUP:
    begin
      syslog(LOG_NOTICE, 'daemon: reloading config');
      // Reload configuration
    end;
  end;
end;

I'll explain the meaning of the procedure HandleSignals later, for now please notice that this routine must be marked with the cdecl calling convention. In short, this is a callback from the system when a signal is received.

Для просмотра ссылки Войди или ЗарегистрируйсяДля просмотра ссылки Войди или Зарегистрируйся

The fork() system call
The fork() system call is used to create processes and returns a Process ID (pid). The fork() call creates a new process, which becomes the child process of the caller and makes (well, the OS does) an identical copy of the address space for the child, so immediately after a fork() call there are two identical copies of address spaces. Remember that from there the two process are running separately (and concurrently).

A very important thing to know is that after the child is created, both processes will execute the next instruction following the fork() call. To tell the difference between the parent from the child we must test the fork()'s pid returned value:

• pid = 0: its the child
• pid > 0: its the parent
• pid < 0: its an error

so, for example, in the following code:

  // Call fork(), to create a child process.
  pid := fork();

  if pid = 0 then
  begin
    // Only the child execute this section
    DoSomething;
  end
  else if pid > 0 then
  begin
    // Only the parent execute this section
    DoSomethingOther;
  end
  else
    raise Exception.Create('Error forking the process');

different sections are executed by the child or the parent based on the returned pid.
Ok, now that we understand the fork() call, let's see how we can use it to create our daemon.


[/SHOWTOGROUPS]

 

[emailx45]

Спойлер: Content - 2nd part 2

[SHOWTOGROUPS=4,20]
The first fork()

begin  
  // syslog() will call openlog() with no args if the log is not already opened
  openlog(nil, LOG_PID or LOG_NDELAY, LOG_DAEMON);

  syslog(LOG_NOTICE, 'before 1st fork() - original process');

  // Call fork(), to create a background process.
  pid := fork();

  syslog(LOG_NOTICE, 'after 1st fork() - the child is born');

  if pid < 0 then
    raise Exception.Create('Error forking the process');

  // Call exit() in the first child, so that only the second
  // child (the actual daemon process) stays around
  if pid > 0 then
    Halt(EXIT_SUCCESS);

  syslog(LOG_NOTICE, 'the parent is killed!');

The first step to create a Linux daemon is the call of the fork() function. The return value is a Process ID (pid), that you can check to see if the fork has been successful. The other important piece is the call of the Halt(EXIT_SUCCESS) function that has the effect of killing the actual process (in this case, the parent) leaving only the child, spawned by the fork call, alive and well (we hope)

If you remember, the child starts executing the instruction right after the fork() so when we read the syslog logfile we'll see 2 lines almost identical, I said almost because the process ID will be different!

As you can see there are two lines: after 1st fork() - the child is born but the pid in one case is 1654 and in the other 1655.

Creating a Unique Session ID (SID)

  // This call will place the server in a new process group and session and
  // detaches its controlling terminal
  if setsid() < 0 then
    raise Exception.Create('Impossible to create an independent session');

  syslog(LOG_NOTICE, 'session created and process group ID set');

Processes in Linux have several IDs, the most important is the Process ID (pid) that identifies the process in the system.

The Session ID exists because processes are organized into sets of sessions and so the child process must get a unique sid from the kernel otherwise, the child process becomes an orphan for the system.

As you can see in the image, the delphid daemon process has the pid of 932 so I can issue this ps command:
ps -Ao pid,ppid,pgid,sid,command | grep -E "COMMAND|932" to show more informations (IDs) about the delphid daemon:

• PID: Process Identifier
• PPID: Parent Process Identifier
• PGID: Process Group Identifier
• SID: Session Identifier (Session Leader)

Linux signals
Signals in Linux are software interrupts and they are used everywhere, so to be integrated with the system, your application needs to handle signals!

  // Catch, ignore and handle signals
  signal(SIGCHLD, TSignalHandler(SIG_IGN));
  signal(SIGHUP, HandleSignals);
  signal(SIGTERM, HandleSignals);

In Linux, every signal has a name (that begins with SIG), the most important (to me) are:

• SIGKILL (signal kill) is sent to a process to request its termination immediately
• SIGINT (signal interrupt) is sent to a process by its controlling terminal when a user wants to interrupt the process. Typically initiated with Ctrl+C
• SIGTERM (signal termination) is sent to a process to request its termination. Unlike the SIGKILL signal, it can be caught and interpreted or ignored by the process
• SIGHUP (signal hang up) is sent to a process when its controlling terminal is closed (originally designed to notify the process of a serial line drop)

Calling the function signal() I choose to handle or ignore the specific signal, in the example you can see that the callback function HandleSignals will be called on the SIGHUP and SIGTERM signals.

The second fork()

  // Call fork() again, to be sure daemon can never re-acquire the terminal
  pid := fork();

  syslog(LOG_NOTICE, 'after 2nd fork() - the grandchild is born');

  if pid < 0 then
    raise Exception.Create('Error forking the process');

  // Call exit() in the first child, so that only the second child
  // (the actual daemon process) stays around. This ensures that the daemon
  // process is re-parented to init/PID 1, as all daemons should be.
  if pid > 0 then
    Halt(EXIT_SUCCESS);
  syslog(LOG_NOTICE, 'the 1st child is killed!');

The second fork() it's only there for a technical reason: ensures that the new process is not a session leader, so it won't be able to regain control of a terminal, since daemons are not supposed to have a controlling terminal. As you can see the code is almost identical to the previous fork().

Close stdin, stdout, stderr file descriptors

  // Close all opened file descriptors (stdin, stdout and stderr)
  for idx := sysconf(_SC_OPEN_MAX) downto 0 do
    __close(idx);

  syslog(LOG_NOTICE, 'file descriptors closed');

File descriptors are inherited to child process, this may cause a waste of resources, so file descriptors should be closed before the fork() call or as soon as the child process starts running.

In a UNIX program there are three special descriptors: the standard input stdin, standard output stdout and standard error stderr and they are tipically used by shell programs, but because a daemon separates itself from the shell, it needs to get rid of everything that's keeping it connected to the terminal.

Redirect stdin, stdout, stderr do /dev/null

  // Route I/O connections to > dev/null

  // Open STDIN
  fid := __open('/dev/null', O_RDWR);
  // Dup STDOUT
  dup(fid);
  // Dup STDERR
  dup(fid);

  syslog(LOG_NOTICE, 'stdin, stdout, stderr redirected to /dev/null');

Well, I said that you have to close stdin, stdout, stderr and now I re-open them!??

The reason is because some library that you are using may try to read or write to/from standard I/O but if you attempt to read/write from a closed file descriptor, the operation will fail. The POSIX specification requires that /dev/null has be provided so the daemon can reasonably depend on this device.

As POSIX assigns descriptors sequentially, __open() call will open stdin and dup() calls will provide a copy for stdout and stderr.

Set file mask and working directory

  // Restrict file creation mode to 640 (750)
  umask(027);
  syslog(LOG_NOTICE, 'file permission changed to 640 (750)');

The umask() (user mask) is a command and a function in POSIX environments that sets the file mode creation mask of the current process which limits the permission modes for files and directories created by the process. In practice with umask() you can define permissions of the new files that your process will create. The user mask contains the octal values of the permissions you want to set for all the new files.

So in order to avoid insecure permissions you want to set the umask() to 027 or to 022.

Set the working directory

  // The current working directory should be changed to the root directory (/), in
  // order to avoid that the daemon involuntarily blocks mount points from being unmounted
  chdir('/');
  syslog(LOG_NOTICE, 'changed directory to "/"');

In a Linux daemon you have to change the current working directory to a safe location, for example the root (/), obviously can be changed later if required.

Changing the working directory allows the system administrator to mount and unmount directories without your daemon getting in the way.

You can also change the working directory to the directory containing all your daemons config/data files, the choice is yours.

Main daemon loop

  syslog(LOG_NOTICE, 'daemon started');
  // deamon main loop
  running := True;
  try
    while running do
    begin
      // deamon actual code
      Sleep(1000);
    end;

    ExitCode := EXIT_SUCCESS;
  except
    on E: Exception do
    begin
      syslog(LOG_ERR, 'Error: ' + E.Message);
      ExitCode := EXIT_FAILURE;
    end;
  end;

  syslog(LOG_NOTICE, 'daemon stopped');
  closelog();

Nothing special with the main loop, this is the point where you have to write your actual code.

Для просмотра ссылки Войди или ЗарегистрируйсяДля просмотра ссылки Войди или Зарегистрируйся
A real example

This is the log of a real Linux service running (in deployment) in an AWS instance, this service act as a Для просмотра ссылки Войди или Зарегистрируйся and Для просмотра ссылки Войди или Зарегистрируйся receiver that trigger some actions based on the activity in the code repository.
For example you can use this in a build machine to build and test your code whenever there is a commit on the repository.

You can find the source code of this example in the Для просмотра ссылки Войди или Зарегистрируйся for this article.

Source code repository

The code is published at Для просмотра ссылки Войди или Зарегистрируйся in this Для просмотра ссылки Войди или Зарегистрируйся. In this repo there are 3 projects:

DelphiDaemonBase
This Для просмотра ссылки Войди или Зарегистрируйся contains the source code showed and discussed in this article, it's a simple but fully functional Linux daemon implementation in a .dpr project with the syslog.pas unit.

DelphiDaemonWiRL
This second Для просмотра ссылки Войди или Зарегистрируйся is built with two (independent) units from the Для просмотра ссылки Войди или Зарегистрируйся:

• WiRL.Console.Posix.Daemon.pas
• WiRL.Console.Posix.Syslog.pas

that contains the (same) code to build a Linux daemon but encapsulated in easy-to-use Delphi classes.

GitHubHooksDaemon
The third Для просмотра ссылки Войди или Зарегистрируйся is a fully featured REST Linux daemon built with the Для просмотра ссылки Войди или Зарегистрируйся and it shows how to encapsulate further the code logic to build a console application that behave as standard console app in debug and a daemon in release.

The demo reacts to valid GitHub WebHooks payloads when a repository is pushed, forked, watched, etc... The interesting thing is that it "restify" GitHub Webhooks calls by converting the X-GitHub-Event extra http header in a REST path in order to manage the calls as standard REST calls.

Conclusions
In this article we learned how to build a proper application that follows the POSIX rules for daemons, we learned how to log in Linux using syslog, a very powerful tool available to Linux developers all built with our beloved development tool: Delphi

If you have a suggestion please contact me or drop a line in the comments below.



[/SHOWTOGROUPS]