12.1 (Conspect) High-level languages and OS

systemd is a main dbus 'follower/consumer'. busctl can show the tree and the introspection looks normal.

The Linux kernel is written in C (because the programming language is macro-assembler):

Also a shell appears not as written but as a gluing(linking) of programs to C++ - for specific libraries and system actions (batch manager)

The list that represents information about Linux: Linux:

• Kernel — C
• Core system libraries — C
• Core system utilities — mostly C and shell
  • Can be C++, Python and Perl
• Scripting and integrating

  • Mostly shell, but Perl was popular and Python is now

  • Sometimes C++

• Almost any OS distribution — all kinds of programming languages

⇒ Is C the only system programming languages?

Windows:

• Core, kernel and libraries: C++ (seldom C)
• Scripting and integrating:
  • Any .Net-based platform (mostly C#)
  • C++
  • Powershell (was: cmd)
• Applications: C++, .Net, almost any 3d-party languages

MacOS:

• Kernel — C

• Core system and libraries — Objective C and C, seldom C++

  • Now: Swift

• Scripting and integration: AppleScript and shell

• Applications: Swift, Objective C, maybe C++, almost any 3d-party languages

The main language of linking is shell, after all. But sometimes there's already a python.

We see that there's a C language everywhere, and there is a C language. a universal system programming language? Yes and no!

Recently, the Rust language has been gaining popularity (C language killer) RUST

Python and OS programming

Any other programming language is suitable for writing system programs with a remark: when we write in C, the fact that the computational model provided to you by the language is extremely close to the hardware on which you program. It makes you treat programming as if it were a stack, registers, allocated memory, heap, call agreements. We have an idea how these bytes are spent, especially important for cores. As soon as we start programming in a high-level language - this thought will disappear (how much and what resources are taken away we don't understand) - we stop thinking like a system programmer, we can make a small utility that eats a lot of memory.

Python: * Cross-platformness (if you look at standard python modules, in some cases they are marked as unix/windows specificity)

• +/-

• ⇒ own implementation of OS features

Example:

   1 import os
   2 import sys
   3 os.getcwd()

* POSIX-oriented (=> Linux)

• +/-

• ⇒ own implementation of non-POSIX OS features

* High-level, but not OS-oriented (unlike shell)

• +/-

• ⇒ own implementation of OS features
• Has a lot of non-privileged syscalls wrappers(The situation where the part is related to the OS and the part is not related)

• Incredible amount of modules at PyPi

  • Including system-oriented
  • Including service-oriented

* Non «resource scrimp» style

• If resources do not multiply, use it as a whole, not piece by peice

  • E. g. .read() instead of .readline()

• no needs to count CPU circles

If you run one of these from windows, there's a whole other way.

OS AND SYS

The os module provides many functions to work with the operating system, and their behavior is usually independent of the OS, so programs remain portable. Here you will find the most commonly used ones.

The sys module provides access to some variables and functions that interact with the python interpreter.

• Cross-platform path: os.path and pathlib

  • .is*(), .exists() etc.

• os:

  • .environ

  • syscalls wrappers (.fork, .getpid, .fstat, .popen, .wait, almost any! …)

• sys: .executable, .argv, .stdin, .stdout, .stderr, …

• Raising a level example: time → datetime → calendar (managing of time and days on the week)

• platform (details of what it is running on)

Subprocess

Allows you to start the process. Concept: cross-platform process execution with communication and exit status control.

   1 import subprocess as proc
   2 proc.run(["ls",  "-al", "/tmp"])

Called the process ls, with parameters.

   1 Res = proc.run([‘ls’,  ‘-al’, ‘/tmp’])
   2 print(Res)

Popen

* High-level popen analog: Popen()

• example

     1 from subprocess import *
     2 # Run first process that outputs to the unnamed pipe
     3 p1 = Popen(["cal -s"], stdout=PIPE)
     4 # Run second process that inputs from the other end of the pipe opened
     5 # and outputs to the second pipe
     6 p2 = Popen(["hexdump", "-C"], stdin=p1.stdout, stdout=PIPE)
     7 # Allow p1 to receive a SIGPIPE if p2 exits
     8 p1.stdout.close()
     9 # Read from the second pipe (stdout, stderr),
    10 # but stderr will be empty because no redirection is used
    11 res = p2.communicate()
    12 # Note data is bytes, not str
    13 print(res[0].decode())
  

Treads

In Python, the trades work "synchronously" (read: not working) due to GIL (global interpreter lock)

• GIL came up with Guido in the last millennium *
• GIL allows very efficient single-stream memory operation
• GIL is as simple as a stick
• GIL doesn't interfere with the multiprocessor. In the case of Python, the differences in performance are small.

The threading module can be used if:

• All but one trades often sleep on I/O
• Algorithm design requires frequent use of common objects

Multitasking

Cross-platform module multiprocessing

• fork() as a basic tool

• Representation of "function" as paralleling units

  • multiprocessing - process-function descriptor

  • process.start() -start the child process

  • process.join() -wait for the child process to complete

• Data exchange through "channels" (more like sockets, because they're bidirectional!)

  • Hole1, Hole2 = Pipe() - two pipe holes

  • HoleA.send() -write data in one end of the pipe (A=1,2)

  • HoleB.recv()  - read data at the other end (B=2,1)

  • Hole.poll() -is there any data to read

• Another data exchange:

  • Очереди (very similar to [[py3doc:queue]).

  • multiprocessing Even more like sockets, including over the network

  • multiprocessing.shared_memory - repository shared memory)

  • and so on.

• Simple task manager multiprocessing - `Pool'.

  • A sequence of handlers (one can be formed) and a sequence of input data for them is formed
  • The dispatcher launches N processes in parallel