SQL Cheatsheet

2023-11-18T00:00:00+00:00

SQL Query Anatomy</h2>
Most databases don't like it when the SQL clauses are out of order.

Operator

`SELECT … FROM …</code> (SQLite</a>)</li>`
`INSERT INTO <table>(…) VALUES (…),…</code> (SQLite</a>)</li>`
`UPDATE <table> SET …</code> (SQLite</a>)</li>`
`DELETE FROM …</code> (SQLite</a>)</li> </ul> </li>`
`JOIN</code></li>`
`WHERE</code></li>`
`GROUP BY</code></li>`
`HAVING</code></li>`
`ORDER BY</code></li>`
`LIMIT</code></li>`
`OFFSET</code></li> </ul>`
NULL Tricks</h2>
Null in SQL is special as it isn't equal (`=</code>) to anything, not even itself, which seems wiered when coming from programming languages but makes perfect sense when considering that NULL</code> means "unknown value" wich definitely isn't equal to another "unknown value".`
`If you want null to behave more like in a programming language use the is</code> keyword, where NULL is NULL</code> evaluates to 1 opposed to NULL = NULL</code> which results in a 0.`
`Note: For all other values is</code> acts like =</code>.`
`To get a default value at the time of a query there is the coalesce()</code> function which takes multiple aruments and returns the first non-null one. (``SQLite</a>)`
`SQLite has a page with more null quirks, comparing SQLite to other Database engines.</a>`
Join Types</h2> An example SQL JOIN</code> statement.</figcaption> SELECT … FROM bookmark INNER JOIN url ON url.url_id = bookmark.url_id …</code></pre> </figure> What all joins have in common is that they glue the tables together and output the entries where the ON</code> condition is true. They differ in how they treat entries that don't make it into the output because they didn't find a "partner" using the ON</code> condition. inner</code> </dt> only returns entries that found a partner. </dd> left</code> </dt> doesn't drop columns from the already present columns, leaving the joined columns NULL</code>. </dd> right</code> </dt> additionally outputs all entries from the joined table that didn't find a partner, leaving the other columns NULL</code>. </dd> full</code> </dt> like a left</code> and right</code> join, returning all of the entries in both tables, joining the ones that are join-able and filling in where joining isn't possible. </dd> </dl> Join Examples</h3> Note: To keep this example as clean as possible types have been omitted, the SQL is intended to be compatible with sqlite3</a>. Use the command sqlite3 :memory:</code> to get an in memory database to play around with. Assume we have two tables, called left_t</code> and right_t</code> for illustration purposes, bot with a single column. CREATE TABLE left_t(lc); CREATE TABLE right_t(rc); INSERT INTO left_t VALUES ('both'),('left'); INSERT INTO right_t VALUES ('both'),('right');</code></pre> Now we can query them an see the results: SELECT lc,rc FROM left_t INNER JOIN right_t ON lc = rc ;</code></pre> We now performed an inner join of right_t</code> against left_t</code> on the single columns having the same value. The returned record is both|both</code>. Pretty intuitive. Now we substitute the INNER</code> for a LEFT</code> and the result is: both|both left|</code></pre> All rows in left_t</code> table mad it through the join, the hole in the right_t</code> was filled with a (blank) NULL</code> field. Note: If the right_t</code> table had more columns all of them would be returned as NULL</code>, independent of any DEFAULT</code> or NOT NULL</code> statements. Similar results for a RIGHT</code> join: both|both |right</code></pre> Now all rows from the right_t</code> table made it through and the missing rows in the left_t</code> table was filled with a NULL</code> entry. And for the FULL</code> join: both|both left| |right</code></pre> All rows preserved, the ones that matched got joined that ones without matches had the resulting holes filled with NULL</code> fields. Database Exploration Commands</h2> You Probably know the problem, a Database you are not familiar with with an engine you don't use every day and what were the commands again? Common Tasks and the Corresponding "SQL" Commands.</caption> Task</th> MariaDB</th> Postgres</th> SQLite</th></tr> Get Help</td> help</code></td> \h</code></td> .help</code></td> </tr> List Databases</td> show databases</code></td> \l</code></td> .databases</code></td> </tr> Use a Database</td> use <database></code></td> \c <database></code></td> Not available</td> </tr> List Tables</td> show tables</code></td> \dt</code></td> .tables</code></td> </tr> Describe Table</td> describe <table></code></td> \d <table></code></td> .schema <table></code></td> </tr> Quit Shell</td> exit</code></td> \q</code>, quit</code>, exit</code></td> .q</code></td> </tr> </table> Git Cheatsheet(s) 2023-11-03T00:00:00+00:00 Setting Up</h2> Before starting to use git one should take the few minutes to set it up. Set up your personal information that will end up on your commits: git config --global user.email <your-mail@example.org> git config --global user.name "<Name>"</code></pre> Set the default branch name to main instead of master The reasons I do this are because there was some controversy around the word "master" in this particular context and and I quickly realised that "main" is way easier to type. git config --global init.defaultBranch main</code></pre> You could set your editor for git commit messages (and other git features) using: git config --global core.editor <editor-command></code></pre> If you (like me) have one editor you want to use for everything: Set the EDITOR</code> variable in your ~/.profile</code></a> and git along with a whole range of other programs will pick up on that. Partial Adding</h2> Situation: You have made changes to one file but they should be in two different commits. git add -p [<file>]</code></pre> git will then chop up the changes into hunks and ask interactively if you want to add/stage (y</code>) a change, ignore/not stage it (n</code>), further subdivide (s</code>) or even edit (e</code>) it. You can use ?</code> to get summary of all options currently available. Undoing Things</h2> Undo last Commit</h3> Situation: You just made a git commit but you committed too much by accident. Note: If you just messed up the commit message or need to add changes use git commit --amend</code>. git reset</code> is your friend here, without any arguments it helps when you git add</code>ed too much, it makes git go into the state it was in at the last commit. git reset HEAD^1</code></pre> The HEAD^1</code></a> resets git to the parent commit of the last commit. Effectively acting as if the last commit never happened. (You can still recover it using the revlog) git reset</code> will only change your git history, not the files you have currently checked out so when using it as described above you won't loose any changes. Warning: git reset --hard</code> is intended to throw changes away and it is possible to loose uncommitted data when using it with the --hard</code> flag enabled. Restore File from last Commit</h3> Situation: You messed up a file and want to throw away the changes since the last commit. git restore <filename></code></pre> This is less destructive than git reset --hard</code> which throws away all your changes in a repository since the last commit. Shallow clone</h2> Situation: You have a very large git repository and limited time and/or bandwidth and only need the latest few commits. git clone --depth 3 <url></code></pre> This example will only fetch the last 3 commits of the main upstream branch. Shallow fetch</h3> In case you need additional branches: git fetch --depth 3 origin <branch> git switch --detach FETCH_HEAD git switch --create <branch></code></pre> After that you can use all branches you fetched as usual Fetching additional Branches</h2> Sometimes you don't get all branches you need with a regular pull, assuming you want to fetch the branch example</code> from origin</code> and use it. # fetch the branch git fetch origin example:example # switch to it git switch example</code></pre> You may have noticed that example</code> is mentioned twice in the fetch command, that is because the syntax is {remote branch name}:{local branch name}</code> where you have the opportunity to rename your local branch should you need it (i.e. when working with multiple upstream repositories). Detecting changes in scripts</h2> To detect any differences of the working tree to the last commit in shellcripts you can use git status --porcelain=v1</code> which outputs a machine readable summary of the changes or nothing at all if there are no changes. To only detect changes to files already checked into git, but not new files or changes you have already staged (TL;DR only changes that git diff would show) for the next commit you can use git diff --quiet</code> which exits with a scess exit code when there are no changes and fail if there are changes. Throwing away changes</h2> To get back to the state that is stored in the last commit you can use the following commands. Warning, destructive commands: These commands arre destructive and you can throw away quite a bit of work with no way to restore it. Use them with care. For a non-destructive version have a look at git stash</code>. To revert files already checked into git to their original state if they were altered or deleted you can use git restore <path></code> where path is a path to a file or whole directory. To remove new files that are not checked in (git stauts</code> shows them as "Untracked files") you can use git clean -f <path></code>, here too, the path argument can point to a file or a directory to operate on. These commands can be useful if you had all changes checked in and a failed build made a mess of your git worktree. Other Peoples Git Guides</h2> These are guides made by other people that I think are useful. In My unorthodox, branchless git workflow</a> Drew DeVault explains how to make use of the fact that the git history of a repository isn't set in stone, I mainly use these tricks to completely avoid a lot of merge conflicts in my personal projects. Learn to change history with git rebase!</a> is Drew DeVault's cheatsheet for altering git history in general and the rebase based workflow in the blogpost liked above. In Dealing with diverged git branches</a> Julia Evans explains how to deal with the situation that changes happend on both the local and remote git repository. (She explains it better than I do here.) Ein kurzer Blick in .git/</a> is a talk held at DiVOC in 2020 by Florian Hars diving into the .git</code> directory, unfortunately no one has translated it to English yet. Inside .git</a> is a blogpost by Julia Evans that also dives into the .git</code> directory and explains what is inside. xdg-mime: Mapping Files to Applications taken apart 2023-10-03T00:00:00+00:00 last time we took apart xdg-open</code>, the freedesktop file-opener and found out that xdg-mime</code> provides the mapping from file to application. So let's take it apart and see what's inside … What can xdg-mime do?</h2> xdg-mime</code> is a shell-script that helps with mapping files to the applications that open them. It implements querying a files mime-type, setting and getting the default application for a given mime-type as well as installing and uninstalling mime-type descriptions. The code is developed in gitlab.freedesktop.org/xdg/xdg-utils</a>. You can use xdg-mime --help</code> or man xdg-mime</code> yourself to get more details. xdg-mime query filetype <file></code> </dt> This sub-command prints the detected mime-type of a given file. (i.e. text/plain</code>) </dd> xdg-mime query default <mimetype></code> </dt> This prints out the name of the desktop file that is configured as the default opener for the given mime-type. (i.e. slatectx-app-kakoune.desktop</code>) </dd> xdg-mime default <applications> <mimetype> ...</code> </dt> Attempts to set the default opener for the one or more given mime-types to the given application which is identified by the name of its desktop file. </dd> xdg-mime install mimetypes-file <file></code> </dt> Installs a mime-type description file that follows the freedesktop mime associations specification</a> where it can be used to identify files. Can be put in user</code> or system</code> mode using the --mode</code> flag. </dd> xdg-mime uninstall mimetypes-file <file></code> </dt> As it says on the tin: It uninstalls a previously installed mime-type file. </dd> </dl> Note: This article ended up being quite long, since it should still be understandable when read in a non-linear manner feel free to skip sections that don't seem interesting. What is in the Script?</h2> Like the other xdg-utils xdg-mime</code> - in addition to a lot of custom logic - imports functionality from the xdg-utils-common.in</code>. Noteworthy imports from the common file are: a way to detect which desktop one is currently using</li> mapping the desktop file to the binary it calls and vice versa</li> helpers for command-line and UI</li> </ul> Unique to xdg-mime</code> is: a callback to update desktop specific mime database caches.</li> querying and setting of default applications from gnome, KDE and generic desktops</li> a function for getting desktop files matching a given mime-type.</li> functionality for figuring out an application when no explicit default has been set</li> functionality for opening the default application for KDE, gnome and generic desktops</li> command line parsing and validating</li> XML parsing in awk … okay</li> lots of parsing and updating files with awk</li> </ul> So this script seems to mostly do the work of parsing and writing the mime to application mapping database, which is to be expected, at first glance mime-type detection seems to be delegated to another command and is in here to provide a stable interface that applications can rely on while xdg-mime</code> does the figuring out how to derive the mime-type. xdg-mime</code> integrates with the KDE and Gnome mime databases, because those have been there before freedesktop.org</a> and the xdg-utils were a thing. More recently Lxqt made their own tool called qtxdg-mat</code></a> that has been integrated into xdg-utils</a>. Actions and CLI Plumbing</h3> Since calling xdg-mime</code> is based on sub-commands, some with their own custom options, this results the parser being a lot of nested case</code> statements and loops for parsing the extra options and arguments. To keep the logic from becoming an indentation mess xdg-mime</code> has a concept of actions. That is, that the command gets translated into an action-name that is written to the action</code> variable. Which is then reused for deciding what to do. Potential values for the action</code> variable are: info</code> </dt> for the filename to mime-type query. with the xdg-mime query filetype</code> sub-command. </dd> makedefault</code> </dt> for setting default mime-handlers with the xdg-mime default</code> sub-command. </dd> defapp</code> </dt> for querying the default application with the xdg-mime query default</code> sub-command. </dd> install</code> </dt> for installing mime descriptions with the xdg-mime install</code> sub-command. </dd> uninstall</code> </dt> for uninstalling mime descriptions with the xdg-mime uninstall</code> sub-command. </dd> </dl> Getting the MimeType of a File</h2> The info</code> action triggers a desktop detection and then decides which mime-type detection method to use. On success the output must be the mime-type of the file followed by a newline without any extra information. If the desktop is KDE and /usr/bin/file</code> is not an executable use info_kde</code>.</li> If the desktop belongs to the gnome gio family (Gnome,Cinnamon,Lxde,Mate,Xfce) use info_gnome</code></li> In case of Lxqt use info_lxqt</code> (new in the beta 1.2.0 release)</li> Otherwise use info_generic</code></li> </ul> Lxqt MimeType Query</h3> For Lxqt qtxdg-mat mimetype <file></code> is used to query the mime information of a given file. KDE MimeType Query</h3> The info_kde</code> checks the KDE_SESSION_VERSION</code> environment variable. For KDE 4 and 6 it uses the kmimetypefinder</code> command, for KDE 5 kmimetypefinder5</code> and for older versions it gets the mime-type from kfile</code>. Extra information is cut away using POSIX tools. Gnome MimeType Query</h3> The info_gnome</code> uses a mechanism similar to file opening where it uses a fallback cascade of multiple methods for detecting the mime type, using the first one that is available. gio info <file></code></li> gvfs-info <file></code></li> gnomevfs-info --slow-mime <file></code></li> </ol> Like with the KDE method any extra information is discarded using standard POSIX tools. Generic MimeType query</h3> The info_generic</code> also uses a little fallback cascade. mimetype --brief --dereference <file></code></li> /usr/bin/file --brief --dereference --mime-type <file></code></li> </ol> For both commands cutting away extra information is not needed. Note on /usr/bin/file</code>: It seems strange that /usr/bin/file</code> is used here instead of just calling the file</code> command. After finding no information on it and asking the original developers (this quirk has been around since 2006</a>) it turns out that nobody knows why the absolute path was used here, it would most probably not make a difference to change it back. How xdg-mime figures out the Default Application</h2> This functionality hides behind the defapp</code> action and can be triggered using the xdg-mime query default <mimetype></code> command. As a reminder this returns the name of the desktop-file describing the application configured as the default opener or an application that seems the best fit according to some simple heuristics in case no default is was explicitly set. This time there are only three implementations, a generic one, a new one for Lxqt and one for KDE below version 6. Note: The application returned by xdg-mime</code> is not necessarily the one that will open when calling xdg-open</code> on the same file</a>, mostly because xdg-open</code> delegates to openers which can have custom logic. KDE Default Application Querying</h3> KDE has a subsystem called Trader</a> which handles delegation of actions between various KDE components (services). To use it xdg-mime</code> invokes ktraderclient</code>, ktraderclient5</code> or ktradertest</code> (for older than KDE 4) as $KTRADER</code>. Asking it for a matching Application</code> service for the given mime-type, extracting the result from the returned text. If no ktrader command appropriate to the current KDE version can be found it falls back to the generic implementation. Lxqt Default Application Querying</h3> Again qtxdg-mat</code> is used: qtxdg-mat defapp <file></code></pre>Generic Default Application Querying</h3> For where there is no desktop specific mechanism, xdg-mime</code> has a generic, specification compliant 🥳 way to get the default application using the defapp_generic</code> function. Desktop Prefixes</h4> Before getting into querying default applications, first imagine someone switches between multiple desktops, maybe mate</code> and kde</code>, while you ave some applications you want to be the default everywhere (i.e. you preferred web-browser) other preferences might depend on the desktop you are currently using like the image- or PDF-viewer. Instead of the desktops stepping on each others toes, every desktop that identifies itself via the XDG_CURRENT_DESKTOP</code> environment variable gets its own configuration file, prefixed by the lowercased name that is read from that variable. This way different preferences on different desktops can be kept separate. This prefix will be referred to as ${prefix}</code> further down. Examples of its contents may be: mate-</code>, kde-</code>, etc. or it may be empty. Note: The XDG_CURRENT_DESKTOP</code> works completely independent of the desktop detection (in fact: the desktop detection uses it). So anyone can set this to get their own configuration space for their fresh new custom desktop. Note: XDG_CURRENT_DESKTOP</code>, like many other XDG variables is a colon separated list that is handled with a fallback cascade. i.e. if your custom xfce</code> based rice identifies itself as my-xfce-rice:xfce</code>, then the application choice will be pulled from my-xfce-rice</code> and automatically fall back to the xfce</code> defaults, even when other desktops are installed. Reading the Association Files</h4> This is a multistage search that … checks the users $XDG_CONFIG_HOME/${prefix}mimeapps.list</code></li> checks the system wide $XDG_CONFIG_DIRS/${prefix}mimeapps.list</code></li> checks again for an applications/${prefix}mimeapps.list</code> in both system and user data directories (usually ~/.local/share/</code> and /usr/share/</code>)</li> searches both user and system data directories for a prefixed with the current desktops name applications/${prefix}defaults.list</code> and applications/${prefix}mimeinfo.cache</code>.</li> falls back to searching the application files for the one that sees itself most fit for opening a file with the given mime-type using the defapp_fallback</code> function</a>.</li> </ul> The files are searched in the order of highest priority to lowest and when the first result is found it is returned. While the filenames are different those files share the same ini syntax.</figcaption> <mimtype>=<desktop-file-name>[;<another-desktop-file-name>...][;]</code></pre> </figure> For the mimeapps.list</code> files, the check_mimeapps_list</code> function searches the [Default Applications]</code> section for an entry with the given mime-type. If such an entry exists, all mentioned desktop-file names are extracted and the first one with a valid executable is returned. If not the search continues. Bug Note: Apparently there is a bug in check_mimeapps_list</code> that discards any names after the first semicolon</a>, making the loop that should handle fallback useless. The defaults.list</code> and mimeinfo.cache</code> get treated simpler than the mimeapps.list</code> files. The first result that mates the mime-type is returned, done. Sections are ignored for these files. Fallback Application Finding</h4> In case no desktop application is found using the configuration file xdg-mime</code> will start searching one itself with the defapp_fallback</code> function. This iterates over all desktop files that match the given mime-type, picking the one with the highest value in InitialPreference</code> (default is 0 when unset). In case multiple applications have the same InitialPreference</code>, the one first found or in the higher priority location wins. If a file is found this way it gets returned. Note: For the InitialPreference</code>, the desktop file specification only says that it is reserved for KDE</a>. Setting Default Applications</h2> Setting a default application for a given mime-type happens in one of two ways: For Lxqt only the make_default_lxqt</code> function gets called. For everything else there are two functions, the make_default_kde</code> one where the KDE database gets updated and make_default_generic</code> which updates the users mimeapps.list</code> file. Weirdly both are always called. (Remember the whole prefix thing?</a>) Generic Default Application Setting</h3> Surprisingly unsurprising the generic case is the simpler one here. It derives the path for $XDG_CONFIG_HOME/mimeapps.list</code> and then runs an awk script on it. This script adds or updates the <mimetype>=<desktop-file-name>;</code> association entry in the [Default Applications]</code> section that belongs to the given mime-type. What is a bit surprising is that the desktop prefix isn't handled here. For Lxqt</h3> Again qtxdg-mat</code> has a sub-command that is a prefect fit and doesn't need any wrapping: qtxdg-mat defapp --set <file> <mimetype></code></pre>For KDE 3 and 4</h3> The make_default_kde</code> function is a lot of awk code that seems to do patching on two files. Unfortunately it hasn't been updated after KDE 4</a> … KDE 4</h4> The KDE 4 specific one modifies a mimeapps.list</code> file in a directory returned by kde4-config --path xdgdata-apps 2> /dev/null | cut -d ':' -f 1</code>. It adds the mime-type to application mapping to the [Added Associations]</code> section if it is missing. My personal guess without doing further research is that this ensures that the application to mime-type mapping exists because something in KDE actually validates that an application can open a given mime-type. I'm also pretty sure that this mechanism relies on the generic setter running immediately after for setting the entry in the [Default Applications]</code> section. Maybe someone can confirm or deny this. KDE 3</h4> For KDE 3 (the default case if the version isn't 4), the profilerc</code> file in a directory returned by kde-config --path config 2> /dev/null | cut -d ':' -f 1</code> gets modified. There with ini-like syntax, a section gets added: [<mimetype> - 1] Application=<desktop-file-name> AllowAsDefault=true GenericServiceType=Application Preference=1 ServiceType=<mimetype></code></pre> Sections that act as duplicates will get removed. Installing and Uninstalling MimeType Descriptions</h2> Now that everything from getting the mime-type to the application is covered one may ask: How does the mime detection tooling know about all the file-types out there? </blockquote> On a freedesktop system this information resides in the XDG Shared MIME-info Database</a> which is a binary compiled from XML files that describe how to detect a given mime-type, and what its icon, names and descriptions (in different languages) are. xdg-mime</code> makes it easier to install those XML descriptions. (When packaging follow the target distributions guidelines first!) For the general case installing puts/deletes the given file in the desired mime/packages</code> directory. For user mode that is $XDG_DATA_HOME/mime/packages/</code>, defaulting to ~/.local/share/mime/packages/</code>. For the system mode $XDG_DATA_DIRS/mime/packages/</code>, defaulting to /usr/share/mime/packages/</code> and /usr/local/share/mime/packages/</code>. There is also a fallback for a KDE3 mechanism called "mimelink" but I will skip that one for now. Note: Don't confuse the files in the packages</code> directory with other XML files named like the mime-types themselves. The ones in packages</code> really are packages containing the descriptions for multiple mime-types while the other ones are already unpacked for better discovery. Hooks on the Mime Database</h3> After (un)installing the update_mime_database</code> function takes care of making sure that the mime database and caches get rebuilt. When in user mode, the currently running desktop is KDE and a display is present kbuildsycoca</code>, kbuildsycoca4</code> or kbuildsycoca5</code> is called, which rebuilds KDE configuration caches. After that every directory in $PATH</code> with the addition of /opt/gnome/bin</code> is searched for a update-mime-database</code> program. All found binaries are ran with the mime database folder that was just updated as the first argument. Note: update-mime-database</code> is the official program for rebuilding the mime database. Relevant Specifications</h2> The XDG Shared MIME-info Database</a> which allows storing information about mime-types themselves.</li> The XDG MIME Applications Associations</a> which are responsible for mapping applications to mime-types.</li> </ul> Until Next Time …</h2> On expectations: When starting to write this article I didn't expect it getting this long and technical, but researching as well as finding and fixing bugs ended up producing a lot of information which I didn't want to hide. A huge thank you goes to Kevin Krammer and Jeremy White for starting the xdg-utils</code> project, to Simon Lees who is currently the de-facto maintainer and everyone who contributed in any way. And while the xdg-utils</code> might be a mess, they are a bloody awesome and quite functional mess! Recently xdg-utils</code> has received a beta release</a> after a very long time. You can help the project by testing it, reporting and researching issues, giving context and writing fixes or documentation. Next up on this mini-series will probably be a dive into xdg-settings</code>, but please don't quote me on that. "It just opens files" - xdg-open under the hood 2023-09-02T00:00:00+00:00 What is xdg-open ?</h2> xdg-open</code></a>, part of the xdg-utils</a> package is a freedesktop.org</a> tool that aims to give you (and everyone else) a desktop independent interface for opening a file or URL, taking care of all the environments quirks and settings so the application or script that "just wants to open a file" doesn't have to. Examples for opening a file, a URI and a file://</code> URI with xdg-open</code></figcaption> xdg-open example.txt xdg-open https://slatecave.net xdg-open file:///usr/bin/xdg-open</code></pre> </figure> This article is based on version 1.1.3</code>, the version that is currently in development is called 1.1.3+</code>. Questions</h2> Before starting out I was trying to find answers to the following questions: How does xdg-open</code> know my favourite applications?</li> Which ways are there to configure it?</li> Is there an interface to extending it?</li> </ul> This is why this article is about high level logic rather than little tricks. First look at the file</h2> To analyse any tool one should try to get it's source code. A quick command -v xdg-open</code> reveals the binary to be under /usr/bin/xdg-open</code> for me, file /usr/bin/xdg-open</code> tells me that it looks like a POSIX</a> shell script</q> and a shellcheck /usr/bin/xdg-open</code> command reveals that file</code></a> was not quite right about the POSIX part. But it is a shellscript that anyone can just open in their favourite text file viewer/editor. The file contains the following: UI code to make it easier to use</li> Helpers for querying .desktop</code> files</li> Lots of checks for a lot of things</li> cli parser code</li> Heuristics for detecting desktop environments</li> Workarounds for tools that seem to misbehave</li> Wrappers for various desktop specific openers</li> Heuristics spaghetti with lots of "xdg" and "generic" in them (I assume those are the really interesting parts)</li> One giant switch that decides what to do</li> </ul> In case you don't have a local copy of a recent xdg-open: Its source lives in the freedesktop gitlab in xdg-open.in</a> and xdg-utils-common.in</a> which is reused across the entire xdg-urils</code> package. This article will link to the 1.1.3</code> tag instead of the latest version. What does it do?</h2> xdg-open</code> really only does one thing, open whatever one throws at it and therefore has a pretty linear flow. It only has options to query help and version information, I'll ignore those code parts. Note: Those options are checked for near the start of the script so a xdg-open --help</code> doesn't even have to load all the logic. Note: There is an undocumented XDG_UTILS_DEBUG_LEVEL</code> environment variable that makes xdg-open</code> more chatty when set to 1</code>. get one URL or filename from the cli arguments</li> try to find out which desktop is currently in use using detectDE</code>.</a></li> fall back to a generic</code> desktop if detection wasn't successful.</li> remove itself from the BROWSER</code> environment variable</li> based on the detected desktop call the appropriate helper for opening the file.</a></li> </ul> Desktops</h2> The "desktops" xdg-open</code> is aware of are (in order of them appearing): kde</code> </dt> Both modern day and very old KDE</a> desktops. </dd> deepin</code> </dt> The Deepin Desktop Environment</a> </dd> gnome3</code> </dt> The GNOME desktop</a> in versions 3 (and 4) </dd> cinnamon</code> </dt> Cinnamon</a>, the Linux Mint</a> Desktop, it is treated as if it was gnome3</code>. </dd> gnome</code> </dt> The GNOME desktop in version 2 </dd> mate</code> </dt> The MATE Desktop Environment</a> that continues as a modern version of GNOME 2, but gets treated differently. </dd> xfce</code> </dt> The Xfce Desktop Environment</a>. </dd> lxde</code> </dt> The Lightweight X Desktop Environment</a>. </dd> lxqt</code> </dt> The Lightweight Qt Desktop Environment</a>, a qt remake of Lxde, but not related enough to share code here. </dd> enlightenment</code> </dt> The Enlightenment</a> Desktop. </dd> </dl> Some of them aren't even desktops: cygwin</code> </dt> We are running under Windows in a Cygwin</a> environment. </dd> wsl</code> </dt> We are running in a WSL environment. Note that this is not yet present in version 1.1.3</code>. </dd> darwin</code> </dt> We are running on MacOS. </dd> flatpak</code> </dt> We are in a flatpak</a> container. </dd> generic</code> </dt> This is a desktop that doesn't require any special treatment 🥳 </dd> </dl> Good news: many of those seem to map to the open_generic</code> function, but for some desktops assumptions are made. Update note regarding Deepin: Previously this article listed deepin</code> as dde</code>, which is icorrect. The XDG-utils use the string deepin</code>. Detecting the desktop</h3> Desktop detection is done by the detectDE</code> function</a> using multiple heuristics in order of decreasing accuracy. The result is written to the DE</code> global script variable. The following tests are executed in order given that all previous tests didn't match: Test if the XDG_CURRENT_DESKTOP</code> environment variable contains a known value.</li> Test for desktop specific clues KDE_FULL_SESSION</code> is set</li> GNOME_DESKTOP_SESSION_ID</code> is set</li> MATE_DESKTOP_SESSION_ID</code> is set</li> The org.gnome.SessionManager</code> name is owned by some application on the d-bus</li> The _DT_SAVE_MODE</code> xprop on the root window contains the string xfce4</code></li> xprop -root</code> has xfce_desktop_window</code> at the start of a line in its output</li> The DESKTOP</code> environment variable begins with Enlightenment</code></li> LXQT_SESSION_CONFIG</code> is set</li> </ul> </li> Test if the DESKTOP_SESSION</code> environment variable contains a known value.</li> Test if uname</code> outputs either CYGWIN</code>, Darwin</code> or Linux</code> In case of Linux, if /proc/version</code> contains the string microsoft</code> and the command powershell.exe</code> is present, assume wsl</code>. (This was added very recently</a> and is not in version 1.1.3</code>)</li> </ul> </li> </ol> The following tests always run: When the desktop is a gnome</code>, test if the gnome-default-applications-properties</code> command is present, if not, assume gnome3</code></li> Test if $XDG_RUNTIME_DIR/flatpak-info</code> is present and if yes assume flatpak</code> (Why isn't this the first check?)</li> </ol> Note: Because this is implemented in xdg-utils-common.in</code> this also applies to all of the other xdg-utils tools. Desktop specific openers</h3> Since xdg-open</code> tries to offload the work when the desktop environment has its own mechanism for opening files lets cover those first. Desktops that "just call an opener"</h4> Desktop</th> Opener</th> Fall back to open_generic</code></th></tr></thead> deepin</td> dde-open</td> yes</td></tr> enlightenment</td> enlightenment_open</td> yes</td></tr> lxqt</td> </td> always</td></tr> cygwin</td> cygstart</td> no</td></tr> darwin</td> open</td> no</td></tr> wsl</td> powershell.exe start</td> no</td></tr> </tbody></table> flatpak</h4> When in a flatpak xdg-open</code> invokes the Desktop</code> portals OpenURI</code> function using gdbus</code></a>. gdbus call --session \ --dest org.freedesktop.portal.Desktop \ --object-path /org/freedesktop/portal/desktop \ --method org.freedesktop.portal.OpenURI.OpenURI \ "" "$1" {}</code></pre> Note: for the 1.1.3+</code> version xdg-open</code> uses the org.freedesktop.portal.OpenURI.OpenFile</code> method for files and file URIs. It also added a timeout of 5 seconds. GNOME, MATE and Xfce</h4> GNOME 2, GNOME 3, MATE and Xfce are related in that they all share a similar cascade of openers (falling back to the next one, if the previous one isn't present) is used. exo-open</code> when on xfce</li> gio open</code></li> gvfs-open</code></li> gnome-open</code> when on gnome</code> (GNOME 2)</li> mate-open</code> when on mate</code></li> open_generic</code></li> </ul> Note: for GNOME 3 the only options really are gio open</code> and gvfs-open</code> before falling back to the generic opener. KDE</h4> For KDE, if the KDE_SESSION_VERSION</code> environment variable is set kde-open</code> is used for KDE 4 and kde-open5</code> for for KDE 5. Otherwise kfmclient exec</code> (with some logic to fix the exit code) is used for even older KDE versions. KDE 6 will use kde-open</code> again</a>, but that isn't in version 1.1.3</code>. Lxde</h4> For Lxde pcmanfm</code> is used if present, but only for filepaths and file://</code> URIs. Otherwise falling back to open_generic</code>. Generic Opener</h2> The generic opener is invoked by the calling the open_generic</code> function</a> with the filepath or URI as the only argument. This is yet another heuristics spaghetti function that tries to delegate work to other tools that may already be configured. Opening files</h3> In case the passed argument looks like a filepath or file://</code> URI, xdg-open</code> tries some additional openers. If the file doesn't exist the script will exit with an appropriate error message. If an X or Wayland display is present (checked using has_display</code>) the files mime-type is determined using xdg-mime query filetype</code> and the function open_generic_xdg_mime</code></a> is attempted for opening. If no display is present or open_generic_xdg_mime</code> returned, opening with run-mailcap</code></a> is attempted if present. run-mailcap</code> apparently is the Debian solution to the problem from before freedesktop standardised file opening. If we have a display and mimeopen</code></a> from the File-MimeInfo perl package</a> is present attempt to open using that. If all of those miss then the open_generic</code> function continues with treating the first argument given to it as if it was an URI. Opening URIs</h3> For URIs the following happens until one of the steps finds a way to open the URI. If there is a display (X or Wayland, checked by has_display</code>) try to derive a URI scheme and open it using open_generic_xdg_mime</code></a> with a mime-type of x-scheme-handler/<scheme></code>. This allows registering custom URI handlers using the already existing mechanism for mime-types. If the BROWSER</code> environment variable (a colon :</code> separated command list) is set try all browser commands in that variable (if %s</code> is present insert the URI in the desired position using printf</code>) until one of them succeeds. (implemented in open_envvar</code></a>) If unset try a set of sane defaults depending on whether a display is present or not. Note: The BROWSER</code> variable was cleaned up at the start of xdg-open</code>. At the time of writing the list of browser commands was … … when a display is present: x-www-browser</li> firefox</li> iceweasel</li> seamonkey</li> mozilla</li> epiphany</li> konqueror</li> chromium</li> chromium-browser</li> google-chrome</li> </ul> … for the terminal (always attempted): www-browser</li> links2</li> elinks</li> links</li> lynx</li> w3m</li> </ul> open_generic_xdg_mime</code></h3> This function</a> takes a filepath (or URI) and a mime-type as input. It uses xdg-mime query default</code> to get the name of the .desktop</code> file, then searches for that in all $XDG_DATA_DIRS/applications/</code>, when found parses the Exec</code> and other needed fields, runs the resulting command and makes the script exit. If it didn't find a matching file it returns. Summary</h2> xdg-open</code> tries to open a file or URI. If it recognises the environment and knows a opener that is specific to that environment</a> it will use that. If there is no specific opener it will fall back to a cascade of generic openers, preferring the freedesktop way, falling back to legacy openers. The freedesktop mime-type to application mapping uses xdg-mime</code></a>. For URIs it will attempt to open the URI as if it was a file with the mime-type x-scheme-handler/<scheme></code>. The last configurable fallback is the BROWSER</code> environment variable which is a colon :</code> separated list of commands of which all will be tried until the first one succeeds. If %s</code> is present in the command the URI or filepath will be inserted in its place instead of being appended. If the BROWSER</code> environment variable is empty, it falls back to a list of hard coded default applications. Extending</h3> For fully custom behaviour wrapping xdg-open</code> still seems to be the best choice.</li> Defaults can be customised by setting a handler in the BROWSER</code> environment variable.</li> xdg-open</code> listens to what xdg-mime</code> says on which mime-type a file is and which application to use for opening.</li> </ul> Expectations</h2> That was quite a bit more desktop-specific stuff in there than expected, while the generic part was supisingly simple at the high level logic view. One thing that was unexpected was that there is no obvious connection to xdg-settings</code></a> i.e. for the default web browser, but maybe that is hidden in xdg-mime</code>. I'll try to follow up with articles on both tools. Make sure you subscribe to my atom feed</a> 😉. Bye! Improving an fzf example with null bytes 2023-07-09T00:00:00+00:00 What is the problem?</h2> On the fzf manpage</a> the --preview-window</code> option is shown off using a combination of git grep</code> for querying every line of every file in the current git repository, fzf</code> for searching and bat</code></a> for a preview with the line currently selected in fzf highlighted and scrolled into the middle of the preview window. (pretty awesome example if you asked me.) The example from the manpage</figcaption> # Preview with bat, matching line in the middle of the window below # the fixed header of the top 3 lines # # […] git grep --line-number '' | fzf --delimiter : \ --preview 'bat --style=full --color=always --highlight-line {2} {1}' \ --preview-window '~3,+{2}+3/2'</code></pre> </figure> The example works by relying on the fact that the output of the git grep</code> looks like <filename>:<linenumber: …</code>, split at the colon, {1}</code> is the filename, {2}</code> is the line number. While the example is very good at explaining how the --preview-window</code> interacts with the --delimiter</code> (its primary purpose), it breaks once there is a colon somewhere in a filename, then {1}</code> and {2}</code> become two halves of an incomplete filename and bat has no idea what it should do with that. Example output produced by git grep --line-number ""</code>. Note that the last line looks a bit funny.</figcaption> test.txt:1:Lorem ipsum test.txt:2:dolor sid … throw:off:1:Does this filename confuse fzf? </samp></pre> </figure> Let's fix it!</h2> The Solution is actually pretty simple: git grep</code></a> supports a -z</code> flag, making it use null bytes instead of colons for separating filename, line number and content, tell fzf to use that as the delimiter using --delimiter '\0'</code> and, the preview isn't confused anymore. One remaining problem is that the menu looks like this: test.txt1Lorem ipsum</samp>, not very readable because those nullbytes are invisible. But there is nothing stopping us from replacing all null bytes with nullbytes followed by a colon and telling fzf to use that combination as a delimiter. git</code> doesn't have something built in for that, but that is what sed</code> was built for. The example from the manpage modified to use null separators in addition to human readable ones.</figcaption> git grep -z --line-number "" | sed -E 's|\x00|\x00:|g' | fzf --delimiter '\0:' \ --preview 'bat --style=full --color=always --highlight-line {2} {1}' \ --preview-window '~3,+{2}+3/2'</code></pre> </figure> In case this looks a bit dull add a --color=auto</code> to the git command and a --ansi</code> to fzf. Done</h2> Given that the more correct version takes noticeably more brain-cycles to decode it is no surprise they used the simple version for the manual. In case you want to: Learn more about null separation in the shell</a>. Happy experimenting! Environment Variable Zoo 2023-07-03T00:00:00+00:00 Environment variables are basically inherited, named arguments to processes with a huge impact on how a Linux or other nix systems work. Reading and Writing Environment Variables</h2> These commands here should work in almost all shells inspired by the original sh, if you use something else you probably know why and how to use it. Read out a single Variable </dt> echo "$SHELL"</code> </dd> printenv SHELL</code> </dd> Dump out all variables </dt> export</code> </dd> printenv</code> </dd> Set one or more variable for a single command </dt> TZ=UTC LANG="de_DE.UTF-8" date</code> </dd> env TZ=UTC LANG="de_DE.UTF-8" date</code> Useful program for simpler shells, environment (un)setting and chain-loading</a>. </dd> Set a variable for the currently running shell </dt> export TZ=UTC</code> </dd> declare -x TZ=UTC</code> </dd> </dl> Persisting Environment Variables</h3> For permanently setting some environment variables there are a number of options (Options further up usually can override the ones below them). .bashrc</code>, .zshrc</code>, etc. </dt> Read whenever you start your favourite shell (as a non-login shell), but don't assume it has effects on the whole session, it is not made for that. </dd> Your Desktop Environment </dt> Desktops like setting environment variables derived from their settings, sometimes for single programs, sometimes for the whole session. </dd> .profile</code> </dt> Read on every login (your login manager is responsible for doing this) and applies to your whole session. </dd> /etc/profile.d/</code> and /etc/profile</code> </dt> System-wide .profile</code> equivalent, remember that you are sharing this one with your packages. </dd> PAM </dt> PAM</a> also sets environment variables. </dd> </dl> The .bashrc</code> (plus friends) and .profile</code> can be reloaded by running source ~/<filename></code> or . ~/.filename</code>. Note: Non POSIX shells usually have their own .profile</code> equivalent, keep that in mind when changing your login shell. Bash for example uses .bash_profile</code> if it exists. Spying on other Processes</h3> Sometimes it is useful to know what environment variables a given running process has currently set, if the process belongs to you or you have root rights (use them responsibly) you have access to the file that contains the process environment. The files path is /proc/$pid/environ</code> (Substitute $pid</code> for the process id), it contains the null separated environment variables. To get them in a pretty way one can use the command tr '\0' '\n' < /proc/$pid/environ</code> (pipe it into the tr command which replaces the null separators with newlines). htop</code> shows the selected processes environment when pressing e</kbd>. Note: This is the reason why it usually is a bad idea to use environment variables for secrets, they can be read out by any other process running under the same user. When transporting secrets from one program to another use pipes and features of your shell to directly read from and write to them. Finding Documentation</h2> Outside of the usual online sources: Read the documentation and read the manpages. Most manpages have an ENVIRONMENT</code> section describing which variables the program reads and/or sets. environ(7) - The manpage about the basics of environment variables. </a> Linux and Posix</h2> You can find a more complete list of the standardised variables in the POSIX (2024, the latest at the time of writing) specification on Environment Variables</a>. HOME</code> </dt> Contains the path to your home directory, as simple as that. (Don't abuse for constructs like $HOME/.config/whatever</code>, that's what the XDG home variables are for.) </dd> PATH</code> </dt> Contains a colon separated list of paths to directories that contain binary files for running them as commands. Directories closer to the front can override others counting binaries of the same name further back. </dd> USER</code> </dt> Should contain the currently logged in username. (Fun fact: the kernel doesn't know your username) </dd> SHELL</code> </dt> Contains the path to the shell binary configured for the current user, this is probably how your favourite terminal emulator knows about your favourite shell without configuration. This may or may not be the shell you are currently using. </dd> TMPDIR</code> or TMP</code> or TEMP</code> </dt> Almost always one of them contains a path to the preferred temporary directory. Useful for when every user gets their own directory with proper permissions. TMPDIR</code> is the one in the POSIX standard and should be the preferred one. </dd> LC_</code>, LANGUAGE</code>, LANG</code> </dt> These contain locale information and are the reason why language settings work across desktop environments. Unfortunately also the reason why you have to restart applications after changing system preferences. For possible values see /etc/locale.gen</code> and /etc/locale.conf</code>. </dd> Set LANG</code> to C</code> for "no translation at all" usually that results in some form of English. </dd> Standardized in the POSIX specification on Internationalization Variables</a>. </dd> TZ</code> and TZDIR</code> </dt> These describe the current timezone to use</a> i.e. Europe/Berlin</code>. Usually points to a tzfile</a> in /usr/share/zoneinfo/</code> </dd> PAGER</code> </dt> A program that can be run using sh -c</code> that shows a textfile (unually in the current terminal) </dd> EDITOR</code> </dt> Like PAGER</code>, but for your favourite text editor. </dd> BROWSER</code> </dt> Like PAGER</code>, but for your favourite browser. (Not standardised anywhere, but a well known convention) </dd> MAILER</code> </dt> Like PAGER</code>, but for sending E-Mail using a program that is compatible with mailx</code></a>, originally intended to allow a custom implementation of /bin/mail</code> on early unixoids. Thank you to inawhilecrocodile for the reasearch!</a> (It was never standardised) </dd> Contrary to what this page previously stated it does not take URLs as argument. </dd> TERM</code> </dt> Contains the name of the terminal, together with the terminfo</a> database the capabilities of the terminal can be found out. When remoting into other machines with a less well known terminal xterm</code> is usually a sane fallback. </dd> MOTD_SHOWN</code> </dt> When set it indicates who has already shown the motd to prevent showing it twice. Convention, probably best known implementation is the pam motd module</a>. </dd> NO_COLOR</code> </dt> When set to a non-empty string Software shouldn't use ANSI-colors unless explicity configured to do so. See no-color.org</a> </dd> </dl> Note: What is meant by run with sh -c</code> is demonstrated pretty well by debians sensible-utils</a>. In a nutshell you lauch your favorite EDITOR</code> using sh -c "$EDITOR \"\$@\"" EDITOR <filepath-goes-here></code> Note: I'm trying to move the BROWSER</code> and MAILER</code> variables to the XDG section. See the related discussion on the xdg mailing list.</a> You'll have to dig around a bit to find all mails 😞. XDG freedesktop.org</h2> freedesktop.org</a> hosts quite a few specifications ensuring interoperability of free (software) desktop envoirnments, these specifications happen to contain quite a lot of envoirnment variables. XDG stands for Cross Desktop Group. Base Directories</h3> These are defines in the Desktop Base Directories Specification</a>, while the below is an overview and you should always read the specification. Note: while one can usually guess where those directories are, it is always a good idea to respect the environment variables instead of blindly writing to some guessed default path, People trying to organise their Home folder will thank you. XDG_DATA_HOME</code> </dt> Path where applications can dump their non-configuration and non-cache persistence. </dd> Default: $HOME/.local/share</code> </dd> XDG_CONFIG_HOME</code> </dt> Path for storing configuration files (and nothing else!) </dd> Default: $HOME/.config/</code> </dd> XDG_DATA_DIRS</code> </dt> colon :</code> separated list of fallback directories for XDG_DATA_HOME</code> containing default data files. (similar to how $PATH</code> works) </dd> Default: /usr/local/share/:/usr/share/</code> </dd> XDG_CONFIG_DIRS</code> </dt> colon :</code> separated list of fallback directories for XDG_CONFIG_HOME</code> containing default configuration files. (similar to how $PATH</code> works). This behaviour is very useful when administrating large desktop systems. </dd> Default: /etc/xdg/</code> </dd> XDG_CACHE_HOME</code> </dt> Path for storing cache files, may contain files specific to the current architecture details, may need cleaning up when on a shared filesystem. </dd> Also be used for files that better get lost rather than ending up in a backup by accident, keepassxc</code> is an example for that. </dd> Default: $HOME/.cache/</code> </dd> XDG_RUNTIME_DIR</code> </dt> For temporary runtime files like sockets, pipes, etc. </dd> Usually: /run/user/<userid>/</code> </dd> </dl> User Directories</h3> There is also the possibility to define User directories, these have the form of XDG_<name>_DIR</code> and are configured using the $XDG_CONFIG_HOME/user-dirs.dirs</code> and taken care of by the xdg-user-dirs-update</code> utility Commonly specified ones are: DESKTOP</code></li> DOWNLOAD</code></li> TEMPLATES</code></li> PUBLICSHARE</code></li> DOCUMENTS</code></li> MUSIC</code></li> PICTURES</code></li> VIDEOS</code></li> SCREENSHOTS</code> - Used by some popular screenshot scripts</li> </ul> Current Desktop</h3> The Freedesktop "Desktop Entry Specification"</a> specifies a XDG_CURRENT_DESKTOP</code> variable that is set from the DesktopNames</code> value in desktops session file (the ones in /usr/share/xsessions/</code> and /usr/share/wayland-sessions/</code> and whatever XDG_DATA_DIRS</code> points to). It contains a colon separated list of names for the current desktop, primarily intended for activating or deactivating desktop entries (i.e. you don't want the gnome settings cluttering up you KDE menus and vice versa). XDG Session</h3> These variables are not part of a specification, the best source for them is a guide for "Writing Display Managers" with systemd</a> and the manpage for the pam_systemd</code> module</a>. Some of them happen to be useful. XDG_SEAT</code> </dt> By definition set to seat0</code>, probably reserved for future use in multiseat scenarios. </dd> XDG_SESSION_TYPE</code> </dt> The best documentation for this one is probably the type</code> argument for the pam_systemd module, roughly it is the display server used for the session. (x11</code>,wayland</code>,mir</code>,tty</code>,unknown</code>) </dd> XDG_SESSION_CLASS</code> </dt> either user</code> or greeter</code> depending on whether the session is primarily logged in or for logging in. </dd> XDG_VTNR</code> </dt> set to the number of the TTY the current session runs in. </dd> XDG_SESSION_ID</code> </dt> Either the /proc/self/sessionid</code> of the auditing session or an "independent counter". </dd> </dl> Graphical Environment</h2> DISPLAY</code> </dt> The id of your X-Display, on your laptop it is probably :0</code>. </dd> WAYLAND_DISPLAY</code> </dt> The name of your wayland socket file in $XDG_RUNTIME_DIR/</code>. </dd> XCURSOR_SIZE</code>, XCURSOR_THEME</code>, XCURSOR_PATH</code> </dt> One way to configure how your cursor looks in most applications. It even works with wayland for when other options are not available. The Arch Wiki has more information on cursor configuration</a>. </dd> DBUS_SESSION_BUS_ADDRESS</code> </dt> Contains information on how to connect to the dbus session bus. Usually set by the dbus-run-session</code> wrapper. </dd> </dl> Mozilla</h2> These are used for Firefox, Thunderbird and other programs that are built on top of Firefox. MOZ_ENABLE_WAYLAND</code> </dt> If set it enables wayland support for all mozilla programs (if not already enabled by default) typically set to 1</code>. </dd> </dl> Java</h2> _JAVA_AWT_WM_NONREPARENTING</code> </dt> Tells java if you have a non-reparenting window manager</a>, usually set to 0 on modern Xorg, to 1 on wayland. If your window stays blank that is an indicator this one is wrong. (Yes the underscore at the start is intentional.) </dd> Possible Values: 0</code>, 1</code> </dd> </dl> More Java environment variables will be added when I run into them. Kubernetes</h2> KUBECONFIG</code> </dt> The path to a yaml file containing connection information for a Kubernetes cluster</a> that tools like kubectl</code></a> and helm</code></a> can use to connect to the cluster. </dd> k3s generates a /etc/rancher/k3s/k3s.yaml</code> file you can point this at. </dd> </dl> Fontconfig 2023-06-20T00:00:00+00:00 What is Fontconfig?</h2> Fontconfig is a library for configuring and customising font access. </blockquote> It is built into almost every program that runs on Linux and has a GUI (even in into old java software!) and makes it very easy to make configure fonts and then forget about it. You can find the Fontconfig documentation over on freedesktop.org</a>. Fontconfig CLI Tools</h2> You average Linux distribution ships Fontconfig along with some commands to poke it. fc-cache</a> </dt> for rebuilding the Fontconfig cache (after installing a font or changing configuration). Usually used like this: fc-cache -f -v</code> </dd> fc-conflist</a> </dt> gives you a list of configuration files found by Fontconfig. </dd> fc-match</a> </dt> for looking up which exact font a given name resolves to, very useful for debugging. </dd> fc-query</a> </dt> to print out metadata for a font file. </dd> fc-validate</a> </dt> can give a hint abut missing glyphs in a font file for a given language. </dd> fc-cat</a> </dt> for having a look at the Fontconfig cache. </dd> fc-list</a> </dt> to list all fonts Fontconfig knows about. </dd> fc-pattern</a> </dt> helps you by parsing Fontconfig patterns</a> , useful for debugging. </dd> fc-scan</a> </dt> similar to fc-query</code> but has the same output format as fc-pattern</code>. </dd> </dl> Font (Family) Names</h2> What one usually calls the fonts name actually is the font family name. This is because of historical reasons from back when a font was a case of lead letters for printing and one needed an extra case full of letters for the bold or italic styles. The family</code> name is usually what is accepted whenever you have some configuration file and it wants a font name. Usually the postscript name works too. Both can be found out using fc-query</code> on a given font file. fc-list</code> also shows the family name by default. Note: fc-query -b</code> omits language and character set information resulting in a more readable output. Note: Fontconfig also has a specific font name</a> that includes the family name, size, style and every other possible font setting. System Fonts</h2> System fonts using Fontconfig are just configuration aliasing generic names to your favourite font. The most common are monopace</code>, sans</code> or sans-serif</code> and serif</code>. If your desktop environment has a font preference dialog in the system settings it is changing which font names those aliases point to. Note: There is also a convention to alias emoji</code> to your preferred emoji font. Finding Fonts that Support a Codepoint</h2> To query fonts that support a given character, you need to know its codepoint. For the following examples I'll use the 😃 emoji which has the codepoint U+1f603</code>. If you have a unicode aware printf</code> command (if you don't know simply try): printf %x "'😃'"</code></pre> Which outputs 1f603</code>. There is also the possibility to use the uni</code> command. uni identify 😃</code></pre> It also lists U+1F603</code>. You could also look it up using a search engine, Wikipedia or one of those Emoji lookup sites. </blockquote> </div> You have the codepoint? Great, now you can ask the fc-list</code> command (replace the example 1f603</code> with the codepoint you are interested in.) fc-list :charset=1f603</code></pre> You should get a list of all fonts on your system that support that codepoint, if that codepoint isn't supported by a font on your system the output stays empty. Configuration Files</h2> You can find your Fontconfig configuration file in $XDG_CONFIG_DIR/fontconfig</code> which in most cases is ~/.config/fontconfig/</code>. In there you can either directly edit the fonts.conf</code> file or a file in the conf.d</code> folder. Global Configuration is in /etc/fonts/</code>. It usually consists of files linked from /usr/share/fontconfig/</code> where configuration snippets for common settings are installed. Using Predefined Configuration</h3> fc-conflist will list all configuration files for fontconfig, the active ones prefixed with +</code> the inactive ones in /usr/share/fontconfig/</code> prefixed with a -</code>. To use the predefined configuration symlink it: ln -s /usr/share/fontconfig/conf.avail/70-no-bitmaps.conf /etc/fonts/conf.d/ # Regenerate the fonconfig cache fc-cache -fv</code></pre> Note: The Void Linux Documentation on Fonts</a> gives a similar example on linking the 70-no-bitmaps.conf</code> to /etc/fonts/conf.d/</code> to mitigate the common issue that Firefox picks pixelated bitmap fonts</a> over higher quality ones. Writing Custom Configuration</h3> To make a valid Fontconfig configuration file it needs some XML scaffolding</a>: <?xml version="1.0"?> <!DOCTYPE fontconfig SYSTEM "urn:fontconfig:fonts.dtd"> <fontconfig>  </fontconfig></code></pre> That scaffold can now be filled with some rules. Note: In case you are in need of an example, have a look at the Fontconfig documentation examples</a> or my Fontconfig</a>. Those rule could be disabling fonts without having to uninstall them, useful when one doesn't have root access or wants to keep the font files for some reason. Example rule for disabling FontAwesome to prevent it from interfering with other icon-fonts.</figcaption> <selectfont> <rejectfont> <pattern> <patelt name="family"><string>FontAwesome</string></patelt> </pattern>  </rejectfont> </selectfont></code></pre> </figure> Another application is to configure the system fonts mentioned before using alias</code> rules. For setting the emoji</code> (or any other) font one simply maps the family name emoji</code> to the family name of the preferred font. An example rule for aliasing emoji</code> to Twemoji</code>.</figcaption> <alias> <family>emoji</family> <prefer> <family>Twemoji</family> </prefer> </alias></code></pre> The same mechanism could be used to alias something like Helvetica</code> to your favourite sans font family.</figcaption> </figure> The prefer</code> part of the alias also accepts multiple font families, this can be used to apply the emoji</code> alias to system fonts by telling Fontconfig to prefer your favourite monospace, sans or serif font and whatever is aliased to emoji</code> over all other installed fonts. <alias> <family>monospace</family> <prefer> <family>Noto Sans Mono</family> <family>emoji</family> </prefer> </alias></code></pre> Of course there are more applications for Fontconfig configurations, but those probably deserve more specific pages. Note: I'm using Twemoji and Noto Sans, not because I like the companies that made them, but because they are under a permissive license are mostly complete, look okay and are one of the very few things those companies are actually doing good for humanity. There are community made fonts like Inter</a> and OpenMoji</a> (plus some more) out there with an impressive coverage and feature set that shouldn't go unmentioned here. Installing fonts</h2> From packages</h3> Usually the packages do everything for you and even install some sane Fontconfig rules. If they don't, have a look at the Fontconfig files they install and check if they clash with your own configuration or if the package description, documentation or comment in its Fontconfig file mentions additional required configuration. This is usually the case for emoji fonts to get them as the default font. From files</h3> In case you have a modern Linux distribution you can install a font by dropping the ttf or otf file into ~/.local/share/fonts/</code> or ~/.fonts/</code>, the latter works on older distributions too. To apply the changes run fc-cache -f -v</code> to rebuild the Fontconfig cache and restart your applications (or log out and back in). Before restarting applications you could verify that the font was installed using the fc-match</code> and fc-list</code> commands. Converting woff2 to ttf</h4> Google (who else 🙄) made a tool for converting woff2 fonts to ttf and back</a>, the package containing it is simply called woff2</code> in most Linux distributions. It contains the woff2_decompress</code> command, give it a path to a woff2 font file and it will convert it to ttf. There is also a woff2_compress</code> that works in the other direction. ANSI Escape Sequences 2023-05-28T00:00:00+00:00 ANSI Escape sequences</a> are some standardised byte sequences that make most Terminals do some thing other than just printing characters on screen. Like printing colourful characters! Using ANSI-Escape Sequences</h2> ANSI-Escape sequences can be used by writing an ASCII-Escape characters followed by some instructions to your Programs standard output or standard error. The codes can be produced by any string escaping mechanism. In proper programming and scripting languages the syntax for the escape character in double quoted strings is usually \x1b</code>. The shell being a bit different you should use either printf</code> or echo -e</code>. While most systems support both the chance of printf</code> being supported is usually higher. Using ANSI-Escape Sequences Responsibly</h2> While colors and text effects are pretty: You should consider the cases where they shouldn't be enabled and respect user choices. Disable color output by default when the NO_COLOR</code> environment variable is non-empty</a>.</li> Disable use of escape codes by default when the output isn't a terminal.</li> Don't overuse color, it gets confusing pretty quickly.</li> Make sure no information gets lost when disabling escape codes. Does using grep on the output work to find information?</li> Does piping it into espeak-ng</code> give a sensible result?</li> </ul> </li> </ul> When writing by default</q> that means unless explicitly requested (i.e. through a --color always</code> CLI option). Seirdy about Best practices for inclusive CLIs</q>. </a> Thank you to Seirdy! Most of the above is a summarized and filtered TL;DR of Seirdys article, you should definitely read it! Setting Text Color and Effects</h2> I'll focus on Select Graphic Rendition (SGR) codes here, Wikipedia has a more complete list</a>. An example for the shell which prints out _FOOBAR_</q> in bright (01</code>) red (31</code>). After that, it resets to default settings (00</code>) and prints a newline character (\n</code>)</figcaption> printf '\033[01;31m_FOOBAR_\033[00m\n' # With %s placeholder printf '\033[01;31m%s\033[00m\n' "_FOOBAR_"</code></pre> </figure> As you can see in the example, multiple escape-codes can be chained in one sequence by delimiting them with a ;</code>. The \033</code> part will turn into an ASCII escape character, the [</code> tells the terminal that this is an ANSI Control Sequence Initiator (CSI) after that can come one or more codes. The sequence is finished by an m</code> to tell the terminal that everything between the CSI</abbr> and it are SGR</abbr> codes. Note on the \033</code> and compatibility: The \033</code> is the octal representation of the Escape ASCII-Character and is the most widely supported one across printf</code> implementations. (The dash printf implementation for example doesn't support hexadecimal representation.) Most commonly used codes</caption> Code</th> Code to reverse</th> Effect</th> </tr> 00</td> </td> Reset all effects</td></tr> Foreground color</td></tr> 30-37</td> 39</td> Set the Foreground color</td></tr> 90-97</td> 39</td> Set the Foreground color, bright variation</td></tr> Background color</td></tr> 40-47</td> 49</td> Set the Background color</td></tr> 100-107</td> 49</td> Set the Background color, bright variation</td></tr> Font effects</td></tr> 01</td> 22</td> Bright text (usually bold and brighter color)</td></tr> 02</td> 22</td> Dim text (usually only effect on color)</td></tr> 03</td> 23</td> Italic text</td></tr> 04</td> 24</td> Underlined text</td></tr> 05</td> 25</td> Blinking text</td></tr> 07</td> 27</td> Reverse foreground and background color</td></tr> 08</td> 28</td> Concealed/Hidden text</td></tr> 09</td> 29</td> Strikethrough text</td></tr> </table> Note: The leading zeros are not needed. Colors</h3> You may have noticed that the 3x</code>, 4x</code>, 9x</code>, and 10x</code> family of codes all use the last digit in the range from 0 to 7 as a color parameter. Table of colors</caption> Code</th> Color</th></tr> 0</td> Black or Dark Grey</td></tr> 1</td> Red</td></tr> 2</td> Green</td></tr> 3</td> Yellow</td></tr> 4</td> Blue</td></tr> 5</td> Violet / Purple</td></tr> 6</td> Cyan or Turquoise</td></tr> 7</td> Light Grey or White</td></tr> </table> Please keep in mind that different people have different color palettes when you want to write reusable scripts. Unobtanium Search 2023-05-27T00:00:00+00:00 What is Unobtanium?</h2> Unobtanium is a stand alone web search engine for making a searchbox for your own little corner of the web. It tries to be easy to self-host and maintain. It doesn't independent on external services and uses SQLite as its database. Why SQLite?: SQLite is great because it doesn't require most of the typical database administration chores, it is simply a file that is managed by unobtanium that requires no special tools for backing it up, moving, etc. Under the hood there is a three step pipeline of crawling, summarising and serving the search frontend. (No feedback loop yet …, though the crawler can remember the last crawl) Th search algorithm is currently a pretty simple "bag of words" type, ranked using bm25 and a custom score that prefers leaf pages over navigation pages</a>. For more information please refer to the technical documentation as it is likely more up to date. Unobtanium Documentation </a> unobtanium.rocks</h2> unobtanium.rocks</a> is the "official" instance that is also a scalability experiment, it has an index focused on small websites, technology, the hacking community (CCC, Freifunk, hackspaces, FabLabs, …), independent wikis, and some other topics in between. The index configuration is public and open to contributions</a>. I want to selfhost!</h2> Great! Though there are a few things to consider … While unobtanium tries to be as easy to use as possible, there are some challenges that are inherent to how the modern web works that one should be aware of. For just a few sites that don't change very often (i.e. a blog) these are almost nonexistent, but they grow with your index and how dynamic the websites are. Selfhosting guide </a> How can I help?</h2> You want to help? Great! You want to write rust code? Help unobtanium on Codeberg</a> it is made of one big repository and multiple smaller parts that help with various aspects of the search engine, any improvement there can almost directly improve unobtanium as a whole.</li> Help with the dependencies! Unobtanium depends on a lot of external crates to do its job. Let me know</a> if you contributed to one because of this 😃.</li> Help with code documentation: Software maintainers are Humans too, they sometimes forget or don't have enough time for good documentation, any improvement there is welcome, even if you "just" edited some comments!</li> </ul> Code reading and markdown? Help with doc.unobtanium.rocks</a></li> </ul> How it Started</h2> Unobtanium started in May of 2023 as a simple web crawler out of curiosity and boredom. The original plan was to create a dead link checker, once that worked the scope was expanded to a simple search application, first with Meilisearch</a> as a backend (around September of 2023), later with SQLite fts5. Shell Null Termination / Separation 2023-05-27T00:00:00+00:00 Null or zero termination or separation is when one uses a null byte to separate records, which is different from the newline usually used in shell pipelines. Why</h2> Usually operating things by a newline is good enough and easy to debug, but even a little shell pipeline like a ls | sort</code> can blow up when it encounters unexpected data like a filename that contains a newline character. Fun fact: Most filesystems actually allow you to use all kinds of characters from emoji to non-printables and control characters. Including newlines. One could even put ASCII-art in a filename (I've already done that)! Using a null character saves you there as it is the only character guaranteed to never occur inside a filename and this becomes more important with other sources of (untrusted) input. The improved version would be ls --zero | sort -z</code> and a | tr '\0' '\n'</code> to convert to newlines for the human that still wants the illusion of line separation. Examples of inputs you shouldn't trust: The human in front of the machine (I know myself)</li> Your filesystem (because USB sticks, downloads, the human, etc.)</li> Anything that comes in over the network (Not even your own service, not the service you are paying for, …)</li> </ul> Translating between Worlds</h2> To translate between newline separated and the null separated world there is a little utility called tr</code></a>. null to newline </dt> tr '\0' '\n'</code> </dd> newline to null </dt> tr '\n' '\0'</code> </dd> swap newline and null </dt> tr '\n\0' '\0\n'</code> </dd> remove potential nulls </dt> tr -d '\0'</code> </dd> </dl> Make sure to actually remove the characters you assume to not be in your input! Getting a Program into Null-Separation Mode</h2> Unfortunately there is no "the one way" to make a program use nulls instead of newlines so here is a a hopefully useful table. Please note that some programs only have the option available in the gnu version, but not in i.e. the busybox version. If a program has some kind of printf</code> option one can use that to make the output null separated. Command</th> Version</th> Input Flag(s)</th> Output Flag(s)</th> </tr> awk</td> gnu, other modern</td> -v 'RS=\0'</td> </td></tr> cut</td> gnu</td> -z</td> -z</td></tr> find</td> </td> </td> -print0</td></tr> fzf</td> </td> --read0</td> --print0</td></tr> grep -r</td> gnu</td> </td> -Z</td></tr> grep</td> gnu</td> -z</td> -z</td></tr> inotifywait</td> </td> </td> --format '…%0' --no-newline</td></tr> ls</td> gnu</td> </td> --zero</td></tr> rg</td> </td> </td> -0</td></tr> sed</td> gnu</td> -z</td> -z</td></tr> sort</td> gnu, busybox</td> -z</td> -z</td></tr> tar -T</td> gnu</td> --null</td> </td></tr> uniq</td> gnu, busybox</td> -z</td> -z</td></tr> xargs</td> gnu, busybox</td> -0</td> </td></tr> read</td> </td> IFS="" read</td> </td></tr> </table> Note: The best way to automatically find out if a program supports an option is to grep -q</code> across the output of its --help</code>. Just make sure to choose a specific enough regex to avoid false positives. Trying out is also an option but that usually is a bit more complex. Note: read</code> is a shell builtin, which is why you don't have to export the IFS</code> (information separator) variable. This also affects other shell builtins like for-loops. Shell Variables and Null Bytes</h2> Shells usually drop null-characters when storing them into variables, so don't rely on that happening and pipe information directly or, if your constraints allow</a>, use bash arrays</a>. Because of this limitation for-loops can't make use of null separation, use while read</code> (will probably result in a subshell though) and xargs</code> instead. Note on IFS</code>: Setting the IFS</code> to a newline by IFS="<newline>"</code> where <newline></code> is a literal line break (as in: the closing quote being the first character of the next line) you at least get clean newline separation if a for loop is really needed. Speech Recognition Frontends 2023-05-09T00:00:00+00:00 This page intends to give an overview what kind of projects out there are open source (or preferably free software) making use of speech recognition without relying on some well known cloud providers. Overview</h2> The projects listed here fall into 3 broader categories. Build your own voice something: voice2json</a></li> Dragonfly</a></li> </ul> Voice Input Methods that could replace Mouse and Keyboard: EasySpeak</a></li> Numen</a></li> Simon</a></li> </ul> Voice assistants: Rhasspy</a></li> Mycroft</a></li> </ul> voice2json</h2> voice2json</code> is a collection of command-line tools</a> for offline speech/intent recognition on Linux. It is free, open source (MIT</a>), and supports 18 human languages</a>. </blockquote> https://voice2json.org/ </a> It can make use of CMU Sphinx</a>, Kaldi</a>, Mozilla DeepSpeech</a> and Julius</a>, depending on the used language profile. Dragonfly</h2> Dragonfly is a speech recognition framework for Python that makes it convenient to create custom commands to use with speech recognition software. </blockquote> https://github.com/dictation-toolbox/dragonfly </a> I learned about this from handsfree.dev</a>. EasySpeak</h2> Voice control for Linux desktops. Fully local, no cloud, Wayland-native. </blockquote> https://github.com/ctsdownloads/easyspeak </a> Tries to be a blend of input method and voice assistant. Numen</h2> Numen is voice control for computing without a keyboard. It works system-wide on Linux and the speech recognition runs locally. </blockquote> https://numenvoice.org/ </a> John Gebbie</a> also has some related projects linked in the README</a>, he also made handsfree.dev</a>. Simon (KDE)</h2> Simon is an open source speech recognition program that can replace your mouse and keyboard. The system is designed to be as flexible as possible and will work with any language or dialect. Simon uses the KDE libraries</a>, CMU SPHINX</a> and / or Julius</a> coupled with the HTK</a> and runs on Windows and Linux. </blockquote> https://simon.kde.org/ </a> Note:Some links have been replaced with their redirect targets. Rhasspy Voice Assistant</h2> Rhasspy (ɹˈæspi) is an open source</a>, fully offline set of voice assistant services</a> for many human languages […] </blockquote> https://rhasspy.readthedocs.io </a> Mycroft Voice Assistant</h2> Mycroft is an open source voice Assistant that makes use of a custom wakeword listener</a> and Mozilla DeepSpeech</a> together with some custom components that has no easy-to-find non-marketing short description. If you are looking for the technical stuff instead of company selling things stuff, you'll find it in the Documentation menu of their page. https://mycroft.ai/ </a> Feedback</h2> If some of the information here is outdated, incorrect or you know a project that is not in this list please open an issue over on Codeberg</a> or contact me in some other way</a>. Scanning, OCR and PDF 2023-04-24T00:00:00+00:00 Scanning</h2> Under Linux one can scan an image with the scanimage</code> command provided by the SANE framework</a>. A helper function that scans an image from the default scanner to the given path with settings suitable for OCR</figcaption> scan_image_to() { printf "Scanning Image …\n" scanimage --mode Gray --format png --resolution 300 -o "$1" -p }</code></pre> Note: The --mode</code> and --resolution</code> options are scanner specific and my work different for you. See scanimage --help</code>. </figure> Unpaper</h3> Unpaper</a> (manpage</a>) is a scan post-processing tool which can remove paper and scan artefacts and also do alignment of carelessly scanned pages. Which is great when preparing them for OCR. Thanks to a friend of mine for mentioning it! OCR using Tesseract and imagemagick</h2> Tesseract</a> is both, a library and command line tool</a> for Optical Character Recognition (OCR). Image in, text in image out. It best works with greyscale images with low background noise and high contrast. Getting a scan closer to that can be done using unpaper or - for scans of known quality - imagemagick using its builtin contrast filter and the level clipping, which is faster than any fully automatic tool. A function that can be used to OCR a given prerotated image that does some preprocessing with imagemagick to improve noise (the -level option) and contrast for scans, the output is the OCR as plaintext to stdout.</figcaption> # Usage: ocr_image <image> ocr_image() { convert "$1" -colorspace Gray -contrast -level 20%,100% -contrast -contrast - | \ tesseract -l deu+eng --oem 2 --psm 3 - - }</code></pre> </figure> To explain the most important tesseract options: -l </dt> To set the used language or script</a>. Accepts multiple values separated by a +</code>. i.e. -l deu+eng</code> </dd> --oem </dt> To set the mode of operation, meaning whether or not to use the LSTM</a> based approach or the original tesseract OCR. At the time of writing I had the best results with option 2, enabling both. </dd> --psm </dt> Tells tesseract what kind of layout analysis</a> to run. I prefer to do a separate OSD (Orientation and Script Detection) only run in mode 0 rotating the image and then using the full automatic layout mode 3 without OSD to do the actual OCR. </dd> --user-words </dt> To give tesseract a text file with a list of words that are likely to occur in the document, this helps avoiding the OCR equivalent to a typo. There is also a pattern file (note that these are not regexes!). More on word and pattern lists in the tesseract manpage.</a> </dd> </dl> Note for Void-Linux users: The package name and command for calling tesseract is tesseract-ocr</code> on Void. This function uses tesseract with the psm 0 mode to correctly orientate a given image file that is assumed to be German or English text. The file is then replaced.</figcaption> # Usage: rotate_scanned_image <image> rotate_scanned_image() { # Replace or leve out the -l option here if you work with other languages/scripts PAGE_OSD="$(tesseract -l deu+eng --oem 2 --psm 0 "$1" - 2>&1)" #if DEBUG_OSD is set print some debug output. [ -n "$DEBUG_OSD" ] && printf "PAGE OSD =====\n%s\n==============\n" "$PAGE_OSD" >&2 APPLY_ROTATION="$( printf "%s\n" "$PAGE_OSD" | awk '/^Rotate:/{ print $2 }' )" convert "$1" -rotate "${APPLY_ROTATION:-0}" "$1" }</code></pre> </figure> Note on DPI settings: Tesseract assumes the images to be scanned at 300dpi, if you are using a different resolution make sure to tell tesseract using the --dpi</code> option. PDF creation and manipulation</h2> Creating a PDF from images</h3> To create a PDF from images without OCR one can use imagemagick. Example that uses the convert</code> command to convert two png files to an pdf with a pagesize of DIN A4.</figcaption> convert -page a4 page_1.png page_2.png document.pdf</code></pre> </figure> Also see the imagemagick documentation on the -page option.</a>. OCR using OCRmyPDF</h3> OCRmyPDF</a> is a python wrapper for tesseract that makes it pretty painless to run OCR on a PDF and turn PDFs into standards compliant PDF/A</a> and optimize PDFs to reduce filesize. An example for OCRing a PDF file that contains German and English text as images.</figcaption> ocrmypdf -l deu+eng input.pdf output.pdf</code></pre> </figure> Package Note: The package is called ocrmypdf</code> for all major Linux distributions (AUR on Arch). Thank you (again) to the friend who found this tool. Creating PDFs with OCR using tesseract and poppler</h3> To create PDFs with an OCR overlay one can delegate the image to PDF step to tesseract using its pdf</code> preset and join the resulting documents after the OCR step with the poppler pdfunite</code></a> tool. Tesseract command to generate a PDF with OCR overlay for a single page with the options left out (they are the same as with the plaintext preset)</figcaption> tesseract page_n.png page_n [options...] pdf</code></pre> </figure> pdfunite command to join multiple PDFs together. Btw. these can be any PDF documents.</figcaption> pdfunite page_.pdf document.pdf</code></pre> </figure> Accessibility Note: These OCR overlays while pretty good are intended for making the documents searchable, they are not a replacement for proper semantic and accessible documents. Getting text out of PDF files</h3> To get text out of a PDF one can use the pdftotext</code></a> and pdftohtml</code></a> command (which come with poppler). pdftotext file.pdf</code></pre>Packages</h2> To get the commandline utilities on some distributions one should also install the <packagename>-bin</code> package. Imagemagick is sometimes just called magick</code>. Tesseract usually comes without the needed data files, these probably come in packages called like tesseract-data-<lang></code>, if you just want everything there is probably a tesseract-data-all</code> metapackage. The pdfunite</code> and pdfto</code> commands are either directly in the poppler</code> package or in poppler-utils</code>. Slatectx: Adding some context to the desktop 2023-04-23T00:00:00+00:00 Why I created slatectx</h2> To put my usecase and workflow in a few sentences: What I'm mostly doing is editing textfiles, writing code, configuration, notes and so on. I'm doing a lot of thing in the shell (bash) and my text editor (kakoune</a>), usually working with more than one window open using i3 or sway as a window manager. Leading to … I want a new terminal and I want it in the same pwd as the last terminal I interacted with on that workspace! </blockquote> The workspace constraint is because I sometimes switch between a reference or otherwise related project to the main thing I'm working on and I don't want the pwd on one to influence the pwd on the other and usually I already separate those by putting them onto a different workspace. Identifying Contexts</h2> To be able to do anything that is context aware, one needs a way to identify a certain context and a way of finding out which context currently applies. Since this journey started with a simple POC and then stayed pretty simple slatectx contexts are named by opaque (no assumptions based on the name) strings containing alphanumeric characters, dashes -</code> and underscores _</code> with slashes \</code> being forbidden because these have to work in filenames. Generating Context Names</h3> With the requirements being that I don't want an additional daemon for doing housekeeping and be able to debug where the names are coming from I settled on the following rough naming scheme: <session_type><sesion_id_or_display>-<workspace></code></pre> For example, the workspace I'm writing this post on maps to the context y1-9</code>, derived from me currently using a wayland session that has wayland-1</code> set as its WAYLAND_DISPLAY</code> and being on workspace 1. With an ssh connection this would simply be ssh</code> and tmux uses t-<tmux_pid></code>. To generate those identifiers my approach involved finding out which session type is currently used</a>, getting the workspace</a> and then stitching that together in the slatectx-get-context-name</code></a> which is the central interface for getting the name of the current context (A bit like xdg-open for opening files). The slatectx-get-context-name</code> script is supposed to be replaced when one wishes to use a different context naming logic. Switching directories</h2> With the problem of figuring out the current context out solved one can implement things like a mot recently used directory (the original motivation). The logic behind that has now moved to its own tool slatectx-pwd</code></a>. When used with the set command it validates that the directory exists (and is a directory), turns a relative into an absolute path and stores the result in $XDG_RUNTIME_DIR/slatectx-pwd/$(slatectx-get-context-name)</code>. The get operation reads this file, runs the checks again (the filesystem could have been modified in between) and prints the path out, depending on the check results with an error return code. This little helper is used in my bashrc</a> to cd</code> into the last used directory on startup and hooked into the PROMPT_COMMAND</code> to record the last used directory. A similar thing happens with a filedialog script</a> I'm using to quickly navigate the filesystem and open files, which on startup gets passed the result of slatectx-pwd get</code> and has a hook to set the pwd on a change-directory event. Another thing that is set up in my shell and text-editor is a cdx</code> alias for quickly jumping to the slatectx-pwd in an already open session. There is also experimental configuration for integrating it with lf</a> which I don't use as often as I'd like to, because I haven't figured out how to make use of lf in a way that makes me not run into a wall at some point where a command I expect isn't there because I haven't figured out how to configure that part of lf yet. (one of the reasons I wrote the filedialog script) Shared text-editor</h2> One thing I missed after switching to kakoune from gedit was that with gedit I could just have a keybind that opened another text editing window and I could just move the tabs between them (not missing the tab UI though). With kakoune having a client-server architecture one could replicate having multiple clients share the same open buffers, if only there was a way to find out what the server is call … yes one can tell kakoune what the socket should be called and whether it should start in client or server mode. That's what the kak-slatectx</code></a> and the kak-e</code></a> are for. kak-slatectx</h3> kak-slatectx</code> replaces the kak</code> command (using an alias) and by default tells kakoune to start with the given slatectx context as the socket name in client or server mode depending on the presence of the socket file. To make it a full replacement, this behaviour is completely disabled, should one attempt to pass the -c</code> or -s</code> options to kakoune. kak-e</h3> kak-e</code> is the slatectx enabled fileopener for kakoune, for it to work kakoune has to remember the last window one interacted with, which is a pretty simple FocusIn</code> hook paired with a custom current_client</code> option</a>. kak-e</code> constructs an edit command for kakoune that then opens the given files in the last focused window, launching a new kakoune instance if needed. Reusing an image viewer</h2> Another thing I set up a wrapper for is imv</a> (which at the time of writing is looking for a new maintainer!), an image viewer which also has a socket for remote controlling it. Since imv always uses its pid for this, an approach similar to slatectx-pwd is used to store the pid of the currently running imv instance and then using that like kak-e</code> to open files in an already open window (Which currently is a bit glitchy). imv-slatectx</code></a> also has an option to disable launching an imv instance, which is useful for preview displays. Open an Image, look at it select the next and the image viewer automatically jumps to the selected image. Close the image viewer and the previews are gone with more screen real estate for managing files. (I've integrated it like this into my lf configuration and it is pretty useful) Conclusion</h2> Here I described why and how I used some relatively simple shellscript to implement some shared pieces of information that can be used to make applications work together. Something I miss with most GUI applications that maybe have a hared recently used, but completely ignore what the one using them is currently working on. That said: Feel free to to reuse my code and ideas on this one and nag me on writing documentation and packaging if you are interested. What is NAT and why you don't want it 2023-03-27T00:00:00+00:00 You may or may not have heard of NAT, Network Address Translation, here is how it works, why you don't want it and why you probably have to use it anyway. How the Internet was supposed to work</h2> The original idea of the internet was that every machine is reachable, can directly connect to other machines using nothing but their public address and also offer serveries. Which worked pretty well until … Why the Internet community started NATing</h3> … the Internet started running out of publicly reachable addresses. So everyone started assigning private IP-Addresses to their internal networks, trying to get the most out of the precious public address space. Legacy NAT</h2> Legacy NAT is simply swapping one IP-Address for another, with a 1:1 relationship between private and public addresses. Either statically assigned or on demand to a machine that wants to connect, so if you wanted to connect to the Internet you have to wait for a free Line (public Address) from a pool of available addresses (called a NAT Pool). Hello old dialup! These are respectively called Static and Dynamic NAT. Modern NAT - Port Address Translation (PAT)</h2> Nowadays with 10s of devices per household with active connections to outside servers that Legacy approach of give a machine an IP when it needs one and recycle after doesn't work anymore. But one can abuse the fact that most machines don't offer services for the public and therefore have a lot of outgoing and no incoming connections. So instead of assigning whole IP-Addresses we made our Routers translate combinations of Layer 4 ports and IP-Address. If one now wants to connect to the server at 1.2.3.4</code> on port 443 from 192.168.0.10</code> from port 12345 via a router assigned 203.0.113.3</code> (it would with multiple public addresses, too) as its public address … The router tries to find a port on its public address using either the original source port or if already in use because someone else behind the NAT had the same idea of a great source port a port near the original port. Let's say we ended up with 12347.</li> Now the router replaces the internal address with the public one and swaps out the port for the publicly reachable one and remembers the combination until the connection closes, there is a timeout.</li> If now an answer arrives on 203.0.113.3</code> port 12345 the NAT knows that the real destination is 192.168.0.10</code> on port 12345 and vice versa.</li> </ul> For ICMP the identifiers can be used similar to port numbers. UDP based protocols need additional support from the NAT as UDP itself has no way of matching requests with replies. With that setup one can support up to ~60000 concurrent TCP connections with a single public IP-Address, great isn't it? Carrier grade NAT</h2> Well, your provider most probably thinks that that is great because now they can serve a whole lot of customers and still only pay for a handful of public IPs. There is even the 100.64.0.0/10</code> network reserved in RFC 6598</a> for exactly this purpose. Problems with NAT</h2> The problem with this is now that your average client is behind two or even more layers of NAT, one at their provider, one built into their home-router. That comes with at least two problems: For real-time applications (Videochat,SIP) you want as direct of a connection as possible which is the opposite of what a NAT does.</li> Long running connections (i.e. your chat or Mail-client waiting for realtime notifications or an ssh session) may get terminated because the NAT needs free ports because 60000 connections is surprisingly little scaling up.</li> </ul> Also NAT actively hinders innovation. Ever heared of SCTP, the Stream Control Transmission Protocol</a> (RFC 9260</a>)? One of the reasons you have probably never seen it in the wild is because old NATs (like Joe Averages router running outdated firmware) have no clue what to do with it. Same problem with QUIC</a> (RFC 9000</a>) which is based on UDP, because punching a hole into the NAT that has the right shape is usually difficult to impossible you probably won't find it in the wild. NAT64</h2> NAT64 was a way to translate IPv6 adresses in the 64:ff9b::/96</code> network to IPv4 but it has been deprecated in favour of dualstack configurations. Probably because for that to work either direct application support is needed or a dns Server which has to translate IPv4 addresses to the NATable IPv6 ones. Myths around NAT</h2> Great I don't need a Firewall!</h3> Yes you do! A NAT only denies incoming connections because no machine in the internal network currently punched a hole in it with UPnP. Your NAT is also waaay too complex for trusting it with denying unwanted connections. (Don't confuse it with a NAT built into a firewall) I want IPv6 NAT because of Privacy!</h3> If you have this kind of network access you can as well reassign random addresses using your favourite IPv6 address assignment mechanism. Solving the NAT problems</h2> So you don't like your NAT anymore? What if the Internet had an address space large enough for every device? Well its your lucky day the solution is called IPv6 and if in 2023 your ISP doesn't give you IPv6, ask them nicely why they are stuck in the past. Building an animated element 2023-03-17T00:00:00+00:00 Accordion elements on the web seem to be everywhere and almost everyone seems to implement them in a way that only works for the well sighted user with a modern browser with JavaScript enabled. I want to show that these can be built in a better way. So why are accordions used at all?</h2> While using accordions or expanders in some situations is a bad idea, sometimes they help with uncluttering, hiding information that isn't needed by most people but available from the already loaded document on request when needed. An example would be to provide contact details on a page, usually visitors don't care, but when they do, that information is easy to find. Another example would be an event log with the summaries always being visible and the details being available inline on request. sourcehut and YunoHost use this pattern. Note: Accordions usually are multiple collapsible elements grouped together and only allow one open at a time. I'm not going into that functionality here. The HTML way of building an accordion …</h2> This pattern is so useful that HTML has elements for exactly that usecase, the <details></code> and <summary></code> elements, one simply puts a summary inside a details tag along with some content and it works, in all browsers, without JavaScript, with any input method the browser supports. Note: Apparently the details element and accessibility is it's own can of worms, too</a>. As the article mentions: You decide. Example of such a details element</figcaption> <details> <summary>Your DNS records for example.org are not configured</summary> <p>Add an <code>A</code> record pointing to <code>93.184.216.34</code>.</p> <p>Add an <code>AAAA</code> record pointin to <code>2606:2800:220:1:248:1893:25c8:1946</code></p> <p>It is important to set both, otherwise peopley may be unable to access your server.</p> </details></code></pre> It then would be displayed similar to this: Your DNS records for example.org are not configured</summary> Add an A</code> record pointing to 93.184.216.34</code>. Add an AAAA</code> record pointing to 2606:2800:220:1:248:1893:25c8:1946</code> It is important to set both, otherwise people may be unable to access your server. </details> </figure> What people are doing instead …</h2> You may have been surprised by the fact that there is an HTML element for exactly this purpose because you have never seen one in the wild. There are probably three reasons people don't use it: They simply don't know that it exists</li> They found out, tried to animate it and failed to do so</li> They use a library which fell victim to one of the previous reasons.</li> </ul> The result being that they give up and write their own accordion from scratch, button onclick</code> handler, element with changing styles, done. Except that the result is usually an accessibility nightmare, and completely falls apart when for some reason the JavaScript needed to operate it isn't available. Why is the details element seemingly hard to animate?</h2> Animating a details tag is simple, one can use the [open=""]</code> CSS selector to find out whether the details element is currently open, put some keyframes and an animation (more details later) on it and it opens with a smooth animation and closing is … not animated at all. No your CSS isn't at fault here. How the details element works.</h3> The details element is a bit of a unicorn in that it manipulates the page tree. When the content gets shown elements are added and when it closes the elements get removed from the page instantly, and while that is great as it saves resources PR is not happy because they want an animation. My personal guess is that this is where most developers who know about the details element give up. Animating it Anyway</h2> So we want an animation for sighted people with modern browsers and a good experience for everyone else, so lets try to achieve that by using the web as it was intended, HTML for content, CSS for eye candy and JavaScript for providing some optional functionality. Note: Animating slightly degrades accessibility (which is probably still better than with your completely custom accordion) of the details element. You can find the final result over on Codeberg.</a> Let's start with the content, I'll assume you have a plain details and summary tag filled with your content and we want to leave it as it is. <details> <summary>Your DNS records for example.org are not configured</summary> <p>Add an <code>A</code> record pointing to <code>93.184.216.34</code>.</p> <p>Add an <code>AAAA</code> record pointin to <code>2606:2800:220:1:248:1893:25c8:1946</code></p> <p>It is important to set both, otherwise peopley may be unable to access your server.</p> </details></code></pre> In case you want to follow along but only have a desktop browser: open a new tab, navigate to about:blank</code> and open up the developer tools. Depending on the browser you now have a pretty good IDE with live preview. The Grand Opening</h3> Before we worry about closing the element, lets worry about it opening. The simplest way here is to simply animate the max-height</code> property from the current height of the element (--details-current-height</code>) to one screen height (100vh</code>), filling only back in time in case the details element is taller than one screen height (i.e. on a smaller device), in that case it will simply snap to full height off screen and is fully viewable. Because we currently have no idea what the current height of the details element is, falling back to an assumed 1em</code> as the height of our summary element works pretty well. (You may have to adjust this based on extra padding and margin you are applying) @keyframes details-appear { from { max-height: var(--details-current-height, 1em); } to { max-height: 100vh; } } details[open=""] { animation: 1s linear details-appear; animation-fill-mode: backwards; /needed for the max-height animation to work/ overflow: hidden; }</code></pre> With that the opening part is taken care of. Fading out of Existence</h3> Because the details element immediately closes and there is no way to animate that we have to put a delay between the actual interaction with the element and the open</code> attribute being removed, so let's add some JavaScript. Because we want to hook on the interaction of a sighted user with the summary element, we use an onclick</code> handler here and assume a details_click_handler()</code> function. The following function simply adds the onclick</code> handler to every summary element on the page (if you only want to target specific elements … I'll leave that as an exercise). function initalize_animated_details() { var details_elements = document.getElementsByTagName("summary"); for ( let i = 0; i < details_elements.length; i++) { details_elements[i].onclick = details_click_handler; } }</code></pre> To run this initalizer you can use any method you like, but since in my experience document.onload</code> is a bit wonky when testing inline code on small documents here is a way to defer it until the first interaction. document.onclick = (event) => { / we only need it once / document.onclick = undefined; initalize_animated_details(); / if the event was for a summary tag run the handler / if (event.target.tagName === "SUMMARY") { return details_click_handler(event); } / Otherwise let the browser handle the click / return true; }</code></pre> With that set up we can now focus on the click handler, which has to do two things: add a style-class that will trigger the animation</li> set a timer to actually close the element after the animation.</li> </ul> A bare minimum implementation of the details_click_handler</code></figcaption> function details_click_handler(event) { let details = event.target.parentElement; / When opening, let the browser do its job / if (!details.open) { return true; } details.classList.add("closing"); setTimeout(function() { details.open = false; details.classList.remove("closing"); }, 500); return false; }</code></pre> Fun fact: if you try that one out right now you'll get the experience of someone who is unable to see the closing animation.</figcaption> </figure> With the bare minimum of JavaScript added let's add some CSS to get a proof of concept animated details element. This is basically the reversed opening from the current height, defaulting to 100vh</code> to the height of the summary element defaulting to 1em</code> again. Again we'll worry about actually setting those variables to something useful later. @keyframes details-disappear { from { max-height: var(--details-current-height, 100vh); } to { max-height: var(--details-summary-height, 1em); } } details.closing { animation: var(--details-close-animation-length, .5s) linear details-disappear; overflow: hidden; max-height: var(--details-summary-height, 1em); }</code></pre> Now you should have a not pretty but animated closing details element. Improving the Animation</h2> Now that we have a proof of concept, lets make it usable. The issues we currently have are: Handle an interaction while the animation is playing correctly.</li> Accessibility impact is larger than it could be.</li> The heights used are still placeholders.</li> The animation speeds don't match up.</li> </ul> Handling an interaction while the animation is playing</h3> Currently when one interacts while the animation is playing the behaviours is that the element keeps closing, which is not great. To fix it we have to add some additional checks to our handler function to test if the closing</code> class is present on an open element. If that's the case we just remove the class gain instead of launching a new timeout and on the timeout part we just discard the timeout if the closing class isn't present. Lowering the impact on accessibility</h3> There are two problems I'm trying to solve with this one: When only relying on something that uses the accessibility interface of the browser (i.e. screenreader only) there is an inexplicable delay between interaction and the element being announced as collapsed, very confusing and/or annoying.</li> When navigating by keyboard, no matter what the output device is, the content inside the details element can still be focused while the animation is playing, again potentially confusing and/or annoying.</li> </ul> Each of the alone would be enough of a reason (at least for me) to not animate the closing of the element on keyboard interaction. (Fixing this is probably possible but at that point we are writing a full blown expander in JavaScript.) Based on this we make a simple assumption: If you are not using a pointing device, you are probably not in the group that greatly benefits from an animation. Note: Of course the "uses a pointer" is a heuristic that will be sometimes be wrong. To achieve this one can test for the offsetX</code> and offsetY</code> values of the event to find out weather the event came from a pointer or not, both are 0 they probably didn't come from a pointer and we tell the browser to do what it would do without us interfering. To stop the opening animation from playing when we won't deliver a corresponding closing animation we add an animated</code> class as an additional requirement in our two animation CSS rules. The offset check to only play animations for events from pointer devices</figcaption> function details_click_handler(event) { let details = event.target.parentElement; if (event.offsetX == 0 && event.offsetY == 0) { details.classList.remove("animated"); return true; } details.classList.add("animated"); // … Rest of the function goes here … }</code></pre> </figure> This way we can make PR and our bosses happy while mostly staying out of the way of people who appreciate clean websites more than fancy animations. Note that after this the animation won't play if the JavaScript isn't running (The element will just snap open and closed like at the start). Deglitching the animation</h3> Now that we have taken care of accessibility issues, we can put the CSS variables introduced earlier to use to make the animation responsive to the current state to avoid some glitches. This means populating the --details-summary-height</code> and --details-current-height</code> properties directly on the details elements. This code is best placed after the offset check so that the properties are already set when opening and closing</figcaption> let summary_height = event.target.getBoundingClientRect().height; let current_height = details.getBoundingClientRect().height; details.style.setProperty('--details-summary-height', summary_height+"px"); details.style.setProperty('--details-current-height', current_height+"px");</code></pre> </figure> Getting the timings to match</h3> After that is fixed the last open issue is, that the speed of the animation opening and the speed of the animation closing don't match. With the opening speed currently being one screen height minus summary per second it is pretty easy to get the closing animation to match that as when the accordion is open we can just ask the browser how tall the details and summary elements are, how tall the screen is and calculate the animation length from that. Since we already know the heights of both relevant elements we'll use the variables from earlier. This code calculating and setting the animation length should be put right before we add the closing</code> CSS class</figcaption> let close_animation_duration = (current_height-summary_height)/window.innerHeight; details.style.setProperty('--details-close-animation-length', close_animation_duration+"s");</code></pre> </figure> To make the actual state match the timeout previously set to 500</code> should now be close_animation_duration1000+10</code> the extra 10ms is to allow the browser a bit of breathing room so that we don't cut the animation off before it has finished. The final result</h3> The resulting details_click_handler</code> function now should look like the following. function details_click_handler(event) { let details = event.target.parentElement; // if called for a non-pointer event don't play animations if (event.offsetX == 0 && event.offsetY == 0) { details.classList.remove("animated"); return true; } details.classList.add("animated"); // tell the animations where to start and where to end let summary_height = event.target.getBoundingClientRect().height; let current_height = details.getBoundingClientRect().height; details.style.setProperty('--details-summary-height', summary_height+"px"); details.style.setProperty('--details-current-height', current_height+"px"); // log the event for debugging // console.log("click", event); // let the browser handle the opening if (!details.open) { return true; } if (details.classList.contains("closing")) { // abort a closing animatiton when interrupted details.classList.remove("closing"); } else { // calculate close animation length let close_animation_duration = (current_height-summary_height)/window.innerHeight; details.style.setProperty('--details-close-animation-length', close_animation_duration+"s"); // trigger closing animation and correponding timeout details.classList.add("closing"); setTimeout(function() { // discard timeout if someone interrupted the animation if ( details.classList.contains("closing")) { // properly close the element and clean up the styleclass details.open = false; details.classList.remove("closing"); } }, close_animation_duration1000+10); } return false; }</code></pre> And our final CSS: @keyframes details-appear { from { max-height: var(--details-current-height, 1em); } to { max-height: 100vh; } } @keyframes details-disappear { from { max-height: var(--details-current-height, 100vh); } to { max-height: var(--details-summary-height, 1em); } } details.animated[open=""] { animation: 1s linear details-appear; animation-fill-mode: backwards; overflow: hidden; } details.animated.closing { animation: var(--details-close-animation-length, .5s) linear details-disappear; overflow: hidden; max-height: var(--details-summary-height, 1em); }</code></pre>Updates</h2> 2023-03-25</h3> Added note about accordions usually having a grouping functionality.</li> Added link to a [great article by Scott O'Hara explaining the problems around how details element is implemented in browsers].</li> Added explanation why I'm disabling the animation for non-pointer inputs here.</li> </ul> IP-Address Ranges 2023-03-08T00:00:00+00:00 In General</h2> While IPv4 and IPv6 have different terminology, they share some concepts. Scope and Delivery schemes. Both IP-Versions share the following delivery schemes: Unicast </dt> one to one, the default </dd> Multicast </dt> one to many, v6 had a huge Update there </dd> Broadcast </dt> usually inside the scope of a single network </dd> </dl> Scope is a bit version dependant, usually you have: Global </dt> These addresses are globally routable and assigned by the IANA</a> and the local Internet registries. </dd> Private </dt> These addresses are reserved for private use and you are free to use the in your own network, just be aware, that you can't use them on the internet. </dd> Loopback </dt> These addresses always route back to the sending machine, usually localhost</code> resolves to them. </dd> Link-Local </dt> These adresses are automatically assigned so that host on the same network can talk to each other. </dd> Documentation </dt> These addresses are reserved for documentation purposes. </dd> Other reserved </dt> These adresses are reserved and have some conditions on when you are supposed to use them. </dd> </dl> IPv6 Adress Ranges</h2> By default IPv6 adresses are unassigned global unicast adresses and therefore reserved for future use. The IANA has a complete list with the IANA IPv6 Special-Purpose Address Registry</a>. Global Unicast (RFC 4291, Section 2.5.4</a>) </dt> 2000::/3</code> </dd> Unique Local (Private) (RFC 4193</a>) </dt> fd00::/8</code> (with Random Global ID) </dd> fc00::/8</code> (reserved for future use) </dd> Loopback (RFC 4291, Section 2.5.3</a>) </dt> ::1/128</code> </dd> Link-Local (RFC 4291, Section 2.5.6</a> </dt> fe80::/10</code> </dd> Documentation (RFC 3849</a>) </dt> 2001:db8::/32</code> </dd> Multicast (RFC 4291, Section 2.7</a>) </dt> ff00::/8</code> </dd> IETF Protocol Assignments (RFC 6890</a>) </dt> 2001:0000::/23</code> </dd> NAT64 Well-Known Prefix (RFC 6052, Section 2</a>) </dt> 64:ff9b::/96</code> </dd> </dl> Note: Site-Local IPv6 Unicast Addresses (RFC 4291, Section 2.5.7</a>) have been deprecated in favour of Unique Local Addresses (RFC 4193</a>) which are pretty similar, but better defined. IPv4 Address Ranges</h2> By default IPv4 adresses are adresses that fall in the global unicast category. The IANA has a complete List with the IANA IPv4 Special-Purpose Address Registry</a>. Private (RFC 1918</a>) </dt> 192.168.0.0/16</code> </dd> 172.16.0.0/12</code> </dd> 10.0.0.0/8</code> </dd> Loopback (RFC 1122, Section 3.2.1.3</a>) </dt> 127.0.0.0/8</code> Note that this is an entire network. </dd> 127.0.0.1</code> is usually used. </dd> Link-Local (RFC 2927</a>) </dt> 169.254.0.0/16</code> </dd> Documentation (RFC 5737</a>) </dt> 192.0.2.0/24</code> (TEST-NET-1) </dd> 198.51.100.0/24</code> (TEST-NET-2) </dd> 203.0.113.0/24</code> (TEST-NET-3) </dd> Shared (RFC 6598</a>) </dt> 100.64.0.0/10</code> used as Shared Address Space to accommodate the needs of Carrier-Grade NAT (CGN) devices.</q> </dd> Benchmarking (RFC 2544</a>) </dt> 198.18.0.0/15</code> </dd> Multicast </dt> 240.0.0.0/4</code> Multicast "Namespace" (RFC 1112</a>) </dd> 192.52.193.0/24</code> Automatic Multicast Tunneling (RFC 7450</a>) </dd> IETF Protocol Assignments (RFC 6890</a>) </dt> 192.0.0.0/24</code> </dd> </dl> Broadcast</h3> Broadcast addresses with IPv4 work by setting the host-part of the address to all ones (in the binary representation) they send to all hosts in the local network. Note: In RFC 919</a> 255.255.255.255</code> is defined as broadcast to all of its immediate neighbors</q>. echoip.slatecave.net 2023-02-26T00:00:00+00:00 Overview</h2> Visit echoip.slatecave.net </a> This service was born out of a combination of wanting to learn rust, not being very happy with the echoip service by mpolden</a> which had some features missing that I wanted in my echoip service. So what echoip-slatecave do? Look up IP-Addresses in a mmdb database and return the resulting geoinformation.</li> Reverse DNS lookup</li> DNS forward lookup for the most common record types, all on one page</li> Has a text view that is made for commandline use</li> Links to other services that may be useful in the HTML version</li> Has a builtin rate-limiter that makes easier to protect a small self hosted instance</li> Interlinked IP-Address and domain lookup to make navigating easier</li> </ul> With that feature set it should be a useful tool where one can enter any kind of address and navigate straight to the needed information or start exploring (which works pretty well, when the IP and domain lookup are interlinked). Why?</h2> But slatian, why? And why include a DNS lookup? I can do this on the commandline! </blockquote> </div> Besides the fact that exploring is easier with a web interface that does multiple lookup with which is one of the reasons I built it … Yes YOU can use the commandline and make sense of it, I can too. But not everyone has the skill or system to use a capable commandline, also I don't want to link to some random non-free wannabe monopoly service because if someone didn't have a tool they'll continue using that. Serial Port Troubleshooting on Linux 2023-02-14T00:00:00+00:00 A serial port (i.e. for programming a microcontroller) can have multiple reasons why it is not working. This guide is based on my experience and will try to cover most cases, FAQ style. Hints</h2> It is always a good idea to try to look up the problem in your distributions wiki/documentation first! You can manually connect to a serial port using screen /dev/tty<name></code>. Quit with Ctrl+a</kbd> and pressing k</kbd> after that. Is your Serial device showing up?</h2> Use ls /dev/tty</code> to get a list of terminal devices on your Linux machine. If you find a ttyUSBn</code> or a ttyACMn</code> (where n</code> is a number) that appears when you plug your device in and disappears when unplugging (yes that's the easiest way) that is your device with hardware and drivers working, skip this section. Note: Builtin serial ports are sometimes also called /dev/Sn</code>. Is the Hardware recognised?</h3> Try running lsusb</code>, it will give you a list of connected USB devices, if your device is not among them you either have a bad cable, a bad microcontroller or (for some micros) the controller is not in programming mode. If the device shows up in lsusb</code> but there is no serial port, your hardware is working but there is a driver problem. (or the hardware doesn't provide a serial port.) Would a reboot fix the problem?</h3> This only the case if there has been a system update since the last reboot. (Depending on your definition of a shortcut: just reboot) If you are not using a very uncommon setup check if /lib/modules</code> (the directory kernel modules (drivers) are in) contains a directory called the same as the output of uname -r</code> (the version of the currently running kernel). If that is not the case your package manager has cleaned up the old modules during an update and you have not rebooted yet (still running the old kernel). The solution in this case is a reboot, then the hardware management will be able to find the modules the kernel needs because the running version matches the version of the modules in the filesystem. Missing Package or Driver Error</h3> If a reboot did not or would not fix your problem try searching for a driver package (usually the package name contains ftdi</code> for USB to serial adaptors). If you find the necessary package already installed, running dmesg -w</code> as root and then plugging in the device is also worth a look at this point. dmesg</code> prints logs from your kernel, including potential information and errors from relevant drivers. Do you have permission for accessing serial ports?</h2> Run ls -l /dev/tty</code>, this will tell you who owns the device files, file permissions apply like with regular files. An example output line, the file belongs to the root</code> user and the dialout</code> group, both are allowed to read and write.</figcaption> crw-rw---- 1 root dialout 4, 67 5. Sep 12:25 /dev/ttyS0 #^ ^ The 2 rw here indicate that owner and group may read from and write to this device.</code></pre> </figure> By default on many distributions the hardware management comes preconfigured in a way that only permits users in the dialout</code> group to use serial ports. Check if you are a member of that group using the groups</code> command. Note: The dialout</code> group may have a different name on some distributions if the ls -l</code> didn't indicate that the serial port belongs to the dialout</code> group this is the case for you. (look up serial ports in the distributions wiki). Note on Arch derived Distributions: On Archlinux and derived distributions the relevant group is called uucp</code> instead of dialout</code>. (UUCP stands for Unix-to-Unix Copy</a> which was common to be used over a serial link.) You can give a user permission by running sudo usermod -a -G dialout <username></code> and logging out and back in after that. In case your system looks completely different you may want to investigate udev</code> and udev rules</a>. Is something else using the port?</h2> Since a program doesn't really know what is on the other side of a serial port some programs simply connect and try to find out. This may interfere with whatever you are trying to do. One of these programs is brltty</a> which is preconfigured by most big distributions. It tries very hard to make sure plugging a braille display into an USB port will make it immediately available. (which is very good, your little Arduino isn't nearly as important!) Unfortunately it has to open every serial device that connects to the system to find out whether it is a braille display or not. To fix that behaviour either configure the "offending" program or stop (and disable) the service it belongs to if you don't need it. Example of how to disable and stop the brltty</code> service on most systemd based distributions. Do not do this if you rely on a working braille display!</figcaption> sudo systemctl disable --now brltty</code></pre> </figure> Did that fix it?</h2> If this guide helped you please consider sharing it. In case you found a mistake or ended up in a dead end, don't hesitate to contact me</a>. arduino-cli 2023-02-13T00:00:00+00:00 While arduino-cli</code> is an awesome tool, unfortunately most guides skip right past it and assume you use the Arduino IDE, this page is a collection of common patterns for arduino-cli</code>. It assumes that you know a few things about how Arduino works. Basics</h2> To create a sketch arduino-cli</code> assumes that your main .ino</code> file has the same name as the directory containing it. To build and upload use arduino-cli compile --fqbn <fqbn></code> and arduino-cli upload -p <serial-device> --fqbn <fqbn></code> in your code directory. Hint: On Linux your serial devices are called /dev/ttyUSBn</code> and /dev/ttyACMn</code> where n</code> is a number. Make sure you have the permissions to access them and nothing else is trying to use those ports.</a> You've got an unhelpful error message? Try adding the --log</code> flag, it will cause arduino-cli</code> to print out whatever it is currently doing. This way one can get some context for the error. FQBN? Fully Qualified Board Name!</h2> The Fully qualified Board name consists of what Arduino calls the Platform ID and a name for the board. You can find it out either by using arduino-cli board search <searchterm></code>, arduino-cli board listall</code> or by appending the official board name in lowercase to the Platform ID. Examples include: arduino:avr:uno</code> (Arduino Uno or another board with an ATMega328P running at 16MHz)</li> esp8266:esp8266:generic</code> (A generic esp8266 Module)</li> esp32:esp32:esp32</code> (A generic esp32 module)</li> </ul> Please note that the esp boards are not available by default. Compile and Upload a Sketch using arduino-cli</h2> I usually create a compile-and-upload.sh</code> script to make development and testing easier. #!/bin/sh arduino-cli compile --fqbn arduino:avr:uno && \ arduino-cli upload --fqbn arduino:avr:uno -p "${1:-/dev/ttyUSB0}"</code></pre> A simple compile and upload script with a fixed board and an optional argument for the serial port that defaults to /dev/ttyUSB0</code></figcaption> </figure> Note: When given the -u</code> flag and the options needed for uploading arduino-cli compile --fqbn arduino:avr:uno -u -p "${1:-/dev/ttyUSB0}"</code> behaves like the above example script. Note on shell syntax: The "${1:-/dev/ttyUSB0}"</code> is read by your shell as replace this with the first argument given to the script or with /dev/ttyUSB0</code>. Adding more Boards</h2> The arduino-cli</code> is not just a wrapper for a compiler and whatever you prefer for uploading … it also is a package manager. If you want more boards (or libraries) you can search for and install them. But if you want to write code for a non-Arduino branded board (i.e. an ESP8266 or ESP32) those packages are not in the Arduino package repositories. Adding an extra repository can be done by letting arduino-cli</code> do the work … arduino-cli config add board_manager.additional_urls 'url://to/index.json'</code></pre> … or by manually editing ~/.arduino15/arduino-cli.yaml</code> and adding the URL pinting to the respositories _index.json</code> file there. board_manager: additional_urls: - https://arduino.esp8266.com/stable/package_esp8266com_index.json - https://dl.espressif.com/dl/package_esp32_index.json - https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json</code></pre> An excerpt from the configuration file from my laptop with additional URL to the stable esp8266 and esp32 repositories in addition to the development version of the esp32 repository.</figcaption> </figure> Don't forget to arduino-cli update</code> and arduino-cli upgrade</code>! Flashing a Bootloader using arduino-cli</h2> arduino-cli</code> already knows how to flash a new bootloader with the burn-bootloader</code> subcommand, you have to tell it for which board you want the bootloader using --fqbn</code> and which programmer you are using with -P</code>. Note: Programmers usually aren't quite cheap, but if you have another Arduino (compatible) with UART (serial) and SPI interface you can use that Arduino as an ISP</abbr> </a>. (use -P arduinoasisp</code>) It may be necessary to provide the port the programmer is on via the -p</code> option. (like with the upload</code> command) Example using the official AVRISP mkII programmer for AVR chips for an ATMEL 328P in an Arduino Uno like configuration</figcaption> arduino-cli burn-bootloader --fqbn arduino:avr:uno -P avrispmkii</code></pre> </figure> To find out which programmer string you need, go to the ArduinoCore-</code> repository of your favourite architecture (See github.com/arduino</a> for the official ones) and open the programmers.txt</code> file (i.e. ArduinoCore-avr/programmers.txt</a>), the keys used there are the needed ids for the -P</code> option. Note: An interesting option might be to use an Arduino Microcontroller as your programmer</a>, use -P arduinoisp</code> in this case. The neccessary code is also in a public repository.</a> Why do I need a Bootloader?</h3> To be able to seamlessly upload code with just a USB to serial converter like in the example above your microcontroller needs to have the Arduino bootloader installed, a small piece of software that runs when the controller starts (that's the reason you need to hook up the reset line when programming) that can be told to write a new program to the controller instead of starting the already existing one. The boards you can buy online come with a bootloader preinstalled. But maybe you want to build your own Arduino compatible or released the magic smoke and want to just buy the chip itself without a development-board, that is when you want to flash the bootloader yourself. Do I need a Bootloader to use the Arduino Framwork?</h3> No, you can use the programmer of your choice (just use the -P</code> option with the upload command) to flash whatever you coded up directly into the controller at the cost of having to use the programmer every time you upload a new program. (Unless that program is a bootloader 😉) Using sketch-local Libraries</h2> To use liberies stored anywhere but the default location (i.e. in a libs</code> folder next to your .ino</code> files) the --library</code> and --libraries</code> can be used. Both accept a comma seperated list of filepaths. --library</code> points to individual libraries and --libraries</code> points to folders of libraries. Example to compile a sketch with a libs</code> folder</figcaption> arduino-cli compile --libraries libs/ --fqbn <fqbn></code></pre> </figure> You may also be interested in</h2> The official arduino-cli documantation </a> Serial Port Troublehooting on Linux </a> Resistor Value Table 2023-01-14T00:00:00+00:00 How To Read Resistors</h2> Color</th> Value</th> Multiplier</th> Tolerance</th></tr> Black</td> 0</td> 1Ω</td> </td></tr> Brown</td> 1</td> 10Ω</td> ± 1%</td></tr> Red</td> 2</td> 100Ω</td> ± 2%</td></tr> Orange</td> 3</td> 1KΩ</td> </td></tr> Yellow</td> 4</td> 10KΩ</td> </td></tr> Green</td> 5</td> 100KΩ</td> ± 0.5%</td></tr> Blue</td> 6</td> 1MΩ</td> ± 0.25%</td></tr> Violet</td> 7</td> 10MΩ</td> ± 0.10%</td></tr> Gray</td> 8</td> </td> ± 0.05%</td></tr> White</td> 9</td> </td> </td></tr> Gold</td> </td> 0.1Ω</td> ± 5%</td></tr> Silver</td> </td> 0.01Ω</td> ± 10%</td></tr> </tbody> </table> Resistors are usually marked with 4 or 5 colored rings to indicate value and tolerance. To read the code in the right direction you have to know that the order is: 1st Ring, 2nd Ring, 3rd Ring (only when you have a 5-Ring code), Multiplier Ring, a larger gap (sometimes not), Tolerance Ring To decode, read the 1st to 3rd Ring as if they were decimals and then multiply that number with the Multiplier. For SMD resistors the pattern apparently repeats itself where the last digit on the package is a Multiplier for the other digits, resulting in a value in ohms. Also the form where there is an R</code> in place of the decimal point is common for smaller values. FFTFM 2022-12-11T00:00:00+00:00 Overview</h2> This script uses fzf</code></a> hooks to make it possible to search with "live" result preview for manpages or RFCs, it also contains some experimental special syntax logic which is supposed to be used for filtering sections of manuals but that part doesn't work very well. It also has issues sorting a particualr manpage to the top when an exact name in typed in (probably due to how apropos</code> works) Note that for the RFCs to work it assumes they are in the place and structure the archlinux rfc</code></a> package places them. Podcasts 2022-11-06T00:00:00+00:00 Note that these are mostly about IT and (I think) well known podcasts. English Language Podcasts</h2> FOSS and Crafts</h3> https://fossandcrafts.org/ </a> FOSS and Crafts is an interdisciplinary exploration of collaborative creation. […] We explore such topics as computer programming, craft production, user freedom and agency, especially informed by the principles of the F(L)OSS (Free, Libre, and Open Source Software) and free culture movements. </blockquote> postmarketOS Podcast</h3> https://cast.postmarketos.org/ </a> You just found the corner of the Internet, where postmarketOS team members discuss whatever crazy development details we encounter while cooking up our favorite mobile Linux distribution. Besides news and anecdotes, once in a while we throw in an interview with an exciting guest who brings us their unique view on the wider mobile Linux scene. </blockquote> Deutschsprachige Podcasts</h2> Chaosradio</h3> https://chaosradio.de/ </a> Chaosradio ist der Podcast des Chaos Computer Club Berlin (CCCB). In dem (bis zu) zweistündigen Format informieren wir seit 1995 über technische und gesellschaftliche Themen. Weiterlesen …</a> </blockquote> C-RadaR</h3> https://www.c-radar.de/ </a> Monatliche Radiosendung des Chaos Computer Clubs auf Radio Darmstadt. Jeden 2ten Donnerstag im Monat, 21-23 Uhr. 103,4 MHz UKW / DAB+ / Stream. Tune In! </blockquote> Pentaradio</h3> https://www.c3d2.de/radio.html </a> Magazin für Netzkultur und Politik des Chaos Computer Clubs Dresden C3D2. jeden vierten Dienstag im Monat 21:30 – 23:00 Uhr (Beschreibung: coloRadio</a>)</cite> </blockquote> Webdirectories 2022-11-03T00:00:00+00:00 Webrings</h2> Listed roughly in order of preference, it is NOT a ranking. Prefernce in this case means: In therms of how many sites I personally conider interesting. Webrings I found more recently or just don't explore as often will naturally rank lower. In case you are new to webrings: Webrings are sometimes members of other webrings and a few sites are members of a lot of webrings so the below are just some possible entry points. Geekring</h3> https://geekring.net </a> For all those funny, quirky, interesting, personal, strange, offbeat websites that's not page 1 on the search engines. </blockquote> Hotline Webring</h3> Has no restrictions for joining and may contain objectionable content. https://hotlinewebring.club/ </a> Geekring with more retro themed websites. The retronaut webring</h3> Has no restrictions for joining and may contain objectionable content. https://webring.dinhe.net/ </a> A webring that contains all kinds of sites. The Yesterweb Ring</h3> https://yesterweb.org/webring/ </a> This webring is for anyone who is tired of how boring and same-y the internet is today. It's for anyone who is sick of seeing websites used purely to drive monetization, informative blogs that ask you to subscribe to see content. </blockquote> XXIIVV Webring</h3> https://webring.xxiivv.com/ </a> This webring is an attempt to inspire artists & developers to build their websites and share traffic amongst each other. The ring welcomes hand-crafted wikis and portfolios. </blockquote> The 512KB Club</h3> https://512kb.club/ </a> The 512KB Club is a collection of performance-focused web pages from across the Internet. </blockquote> More of a directory but sometimes considered a webring (it's missin the ring-part though) envs - webring</h3> https://envs.net/ring/ </a> this webring can be joined by any user on envs.net. </blockquote> Actual web directories</h2> The real phonebooks of the internet! ooh.directory</h3> https://ooh.directory/ </a> A collection of 2,295 blogs about every topic </blockquote> The counter in the description is probably outdated by the time you are reding this 😃. Sites with lots of interesting links</h2> Seirdy’s Home</h3> https://seirdy.one/ </a> Not only a source of great links but also great articles containing them. TinyGem</h3> https://tinygem.org/ </a> Have you ever bookmarked a page and worried that you will not find the same stuff next time you are there? Keep the pages on the web saved forever. </blockquote> A searchable link aggregator for links people find interesting. ASCII Table 2022-10-16T00:00:00+00:00 This ASCII-Table is mostly derived from the Unicode 15.0 Documentation</a>. Mostly Printable Characters</h2> Skip to the control characters table </a> Codes 0x20 to 0x7E are referred to as the printable characters, 0x7F is a control character. Wikipedia has more background information.</a> Numbers and Specials</caption> Char</th> Hex</th> Dec</th></tr> SPACE</td> 20</td> 32</td></tr> !</td> 21</td> 33</td></tr> "</td> 22</td> 34</td></tr> #</td> 23</td> 35</td></tr> $</td> 24</td> 36</td></tr> %</td> 25</td> 37</td></tr> &</td> 26</td> 38</td></tr> '</td> 27</td> 39</td></tr> (</td> 28</td> 40</td></tr> )</td> 29</td> 41</td></tr> </td> 2a</td> 42</td></tr> +</td> 2b</td> 43</td></tr> ,</td> 2c</td> 44</td></tr> -</td> 2d</td> 45</td></tr> .</td> 2e</td> 46</td></tr> /</td> 2f</td> 47</td></tr> 0</td> 30</td> 48</td></tr> 1</td> 31</td> 49</td></tr> 2</td> 32</td> 50</td></tr> 3</td> 33</td> 51</td></tr> 4</td> 34</td> 52</td></tr> 5</td> 35</td> 53</td></tr> 6</td> 36</td> 54</td></tr> 7</td> 37</td> 55</td></tr> 8</td> 38</td> 56</td></tr> 9</td> 39</td> 57</td></tr> :</td> 3a</td> 58</td></tr> ;</td> 3b</td> 59</td></tr> <</td> 3c</td> 60</td></tr> =</td> 3d</td> 61</td></tr> ></td> 3e</td> 62</td></tr> ?</td> 3f</td> 63</td></tr> </table> Uppercase</caption> Char</th> Hex</th> Dec</th></tr> @</td> 40</td> 64</td></tr> A</td> 41</td> 65</td></tr> B</td> 42</td> 66</td></tr> C</td> 43</td> 67</td></tr> D</td> 44</td> 68</td></tr> E</td> 45</td> 69</td></tr> F</td> 46</td> 70</td></tr> G</td> 47</td> 71</td></tr> H</td> 48</td> 72</td></tr> I</td> 49</td> 73</td></tr> J</td> 4a</td> 74</td></tr> K</td> 4b</td> 75</td></tr> L</td> 4c</td> 76</td></tr> M</td> 4d</td> 77</td></tr> N</td> 4e</td> 78</td></tr> O</td> 4f</td> 79</td></tr> P</td> 50</td> 80</td></tr> Q</td> 51</td> 81</td></tr> R</td> 52</td> 82</td></tr> S</td> 53</td> 83</td></tr> T</td> 54</td> 84</td></tr> U</td> 55</td> 85</td></tr> V</td> 56</td> 86</td></tr> W</td> 57</td> 87</td></tr> X</td> 58</td> 88</td></tr> Y</td> 59</td> 89</td></tr> Z</td> 5a</td> 90</td></tr> [</td> 5b</td> 91</td></tr> \</td> 5c</td> 92</td></tr> ]</td> 5d</td> 93</td></tr> ^</td> 5e</td> 94</td></tr> _</td> 5f</td> 95</td></tr> </table> Lowercase</caption> Char</th> Hex</th> Dec</th></tr> `</td> 60</td> 96</td></tr> a</td> 61</td> 97</td></tr> b</td> 62</td> 98</td></tr> c</td> 63</td> 99</td></tr> d</td> 64</td> 100</td></tr> e</td> 65</td> 101</td></tr> f</td> 66</td> 102</td></tr> g</td> 67</td> 103</td></tr> h</td> 68</td> 104</td></tr> i</td> 69</td> 105</td></tr> j</td> 6a</td> 106</td></tr> k</td> 6b</td> 107</td></tr> l</td> 6c</td> 108</td></tr> m</td> 6d</td> 109</td></tr> n</td> 6e</td> 110</td></tr> o</td> 6f</td> 111</td></tr> p</td> 70</td> 112</td></tr> q</td> 71</td> 113</td></tr> r</td> 72</td> 114</td></tr> s</td> 73</td> 115</td></tr> t</td> 74</td> 116</td></tr> u</td> 75</td> 117</td></tr> v</td> 76</td> 118</td></tr> w</td> 77</td> 119</td></tr> x</td> 78</td> 120</td></tr> y</td> 79</td> 121</td></tr> z</td> 7a</td> 122</td></tr> {</td> 7b</td> 123</td></tr> |</td> 7c</td> 124</td></tr> }</td> 7d</td> 125</td></tr> ~</td> 7e</td> 126</td></tr> DEL</td> 7f</td> 127</td></tr> </table> </div> Note on the DEL</code> code:The DEL being 0x7F, meaning it has all 7 bits set, probably originates from how one "deleted" something from paper tape, by just punching all of the holes one could "erase" the original character. Its meaning is ambigious, but it usually refers to a backspace. Fun Fact: The lowercase characters in ASCII were an "aftertought" and only added later in 1965, two years after the initial release. Control characters</h2> Char</th> Hex</th> Oct</th> Dec</th> Name</th> C Esc</th></tr> NUL</td> 00</td> 000</td> 0</td> Null</td></tr> SOH</td> 01</td> 001</td> 1</td> Start Of Heading</td></tr> STX</td> 02</td> 002</td> 2</td> Start Of Text</td></tr> ETX</td> 03</td> 003</td> 3</td> End Of Text</td></tr> EOT</td> 04</td> 004</td> 4</td> End Of Transmission</td></tr> ENQ</td> 05</td> 005</td> 5</td> Enquiry</td></tr> ACK</td> 06</td> 006</td> 6</td> Acknowledge</td></tr> BEL</td> 07</td> 007</td> 7</td> Bell / Alert</td> \a</code></td></tr> BS</td> 08</td> 010</td> 8</td> BAckspace</td></tr> HT</td> 09</td> 011</td> 9</td> HOrizontal Tab</td> \t</code></td></tr> LF</td> 0a</td> 012</td> 10</td> Line Feed</td> \n</code></td></tr> VT</td> 0b</td> 013</td> 11</td> Vertical Tab</td> \v</code></td></tr> FF</td> 0c</td> 014</td> 12</td> Form Feed</td> \f</code></td></tr> CR</td> 0d</td> 015</td> 13</td> Carriage Return</td> \r</code></td></tr> SO</td> 0e</td> 016</td> 14</td> Shift Out</td></tr> SI</td> 0f</td> 017</td> 15</td> Shift In</td></tr> DLE</td> 10</td> 020</td> 16</td> Data Link Escape</td></tr> DC1</td> 11</td> 021</td> 17</td> Device Control 1</td></tr> DC2</td> 12</td> 022</td> 18</td> Device Control 2</td></tr> DC3</td> 13</td> 023</td> 19</td> Device Control 3</td></tr> DC4</td> 14</td> 024</td> 20</td> Device Control 4</td></tr> NAK</td> 15</td> 025</td> 21</td> Negative Acknowledge</td></tr> SYN</td> 16</td> 026</td> 22</td> Synchronous Idle</td></tr> ETB</td> 17</td> 027</td> 23</td> End Of Transmission Block</td></tr> CAN</td> 18</td> 030</td> 24</td> Cancel</td></tr> EM</td> 19</td> 031</td> 25</td> End Of Medium</td></tr> SUB</td> 1a</td> 032</td> 26</td> Substitute</td></tr> ESC</td> 1b</td> 033</td> 27</td> Escape</td> \e</code></td></tr> FS</td> 1c</td> 034</td> 28</td> File Seperator</td></tr> GS</td> 1d</td> 035</td> 29</td> Group Seperator</td></tr> RS</td> 1e</td> 036</td> 30</td> Record Seperator</td></tr> US</td> 1f</td> 037</td> 31</td> Unit Seperator</td></tr> </table> Note: The Names have been titlecased to make them easier to read. Notation</h2> Escaping in Scripting and Programming</h3> Sometimes in languages one can't use all the characters available in the source file, especially when using control characters. Encoding these as other characters is called escaping. This usually happens with text between double quotes "</code>. There are differences between languages and implementations but the general rules are: Special characters preceeded by a backslash \</code> are taken literally. (i.e. \"</code> or \\</code>) (The exact behaviour varies a lot by language.)</li> Latin characters preceeded by a backslash to generate control characters (i.e. \n</code>) (The C Esc column)</li> Hexadecimal value of the character prefixed by \x</code> (i.e. 0x1b</code>) (should work almost everywhere)</li> Octal encoding prefixed by just a backslash (i.e. \033</code>) (Use this when using printf</code> on the command line</a>)</li> </ul> Note: C Esc</code> here is short C-Escape as the C language apparantly started this way of excaping characters in strings. It is now used by almost all modern languages and in other contexts. i.e. printf</code>, sed</code>, etc. exact support may vary. Escaping in XML and HTML</h3> XML and html have their own way of escaping non-printable characters, this involves a sequence sandwiched between an ampersand &</code> and a semicolon ;</code>. In general Characters can be escaped using &#<dec>;</code> where <dec></code> is replaced by the decimal value associted with the character. (i.e, &</code> to encode an ampersand &</code>) There are also named escapes to make remembering them easier: Character</th> XML-escape</th></tr> &</code></td> &</code></td></tr> <</code></td> <</code></td></tr> ></code></td> ></code></td></tr> "</code></td> "</code></td></tr> </table> Control and Shift</h3> With the ASCII table control and shift keys an be implemented using simple addition and substraction. The lowercase character can be obtained by adding 32 (0x20) to the code of the corresponding uppercase character and the control key goes 64 (0x40) in the opposite direction. With that ctrl+c</kbd> maps to code 3 "End of Text". And ctrl+d</kbd> maps to "End of Transmission". You may know those shortcuts from the terminal, they hopefully make a bit more sense now. Control characters are sometimes written down/printed as the character one gets when adding 64 to the control characters value prefixed by an ^</code>. This maps escape to ^]</code> and the nullbyte to ^@</code>. (You have probably seen those when opening a binary file in a text editor.) Unicode</h2> Creating a unicode table andkeeping it updated is out of scope here, besides that: Wikipedia has a List of Unicode Characters</a> and there are the offical Unicode Character Code Charts</a>. (Plus a whole lot of other unicode tables out there.) Notation Hint: Unicode codepoints (that is the number assigned in the unicode standard, not the binary character representation) are written U+<hex></code> where <hex></code> is the codepint number in hexadecimal. Example: U+1f603</code> for 😃. There is also a little commandline tool called uni</code></a>, that is pretty good at providing a searchable unicode table. Shell Snippets 2022-10-16T00:00:00+00:00 The snippets on this site are intended for getting copied and being useful so please do that. Argument Parser</h2> A simple argument parser for bash. #!/bin/bash # Make the script fail when a command fails set -e # Settings variables go here FOO="foo" show_help() { cat <<EOF Usage: foo <<options>> <bar> Blubber blah blah ... EOF } while [ "$#" -gt 0 ]; do case "$1" in --foo) FOO="$2" shift 2;; --help) show_help; exit 0;; ) printf "Unknown option: %s\n" "$1"; exit 1;; esac done # Logic goes here echo "$FOO"</code></pre>set -e</code> and shift</code></h3> The set -e</code> option tells the shell to stop executing a script once a command fails. This is important when using a shift</code> in this kind of while loop where the loop exiting depends on the shift</code> working. Example: The shift may fail if it is told to move 2 arguments out (shift 2</code>) but there is only one given. In the example above that would be fulfilled when passing --foo</code> as the last argument (a relatively easy to make mistake). If the shift silently failed the argument counter would never reach 0 and the script would be stuck in an endless loop. mktemp --tempdir</code> but for not just gnu</h2> With the GNU and busybox coreutils the --tempdir</code> option of mktemp is very useful when using mktemp</code> with a template name and you want it to stay in the standard tempdir. The following snippet also does this for other mktemp</code> implementations: mktemp -p "${TEMPDIR:-/tmp/}" your-template-here.XXXXXXXXX</code></pre>Hook Helper</h2> This little function calls its argument as an ad-hoc shellscript if it is non-empty, if it returns a nonzero, one should call the default action. # on fail do default action run_hook() { [ -n "$1" ] && bash -c "$1" }</code></pre> Example of how this little helper is intended to be used</figcaption> export VARIABLE_WHICH_IS_USEFUL_FOR_HOOK="something" run_hook "$HOOK_COMMAND" || default_action</code></pre> </figure> An example of this being used</a> can be found in my dotfiles with the filedialog. (Fun fact: It was created for this script) Grep Regex Match Helper</h2> This function pipes its first argument into a grep -q</code> command which is given the rest of the arguments. matches() { TEXT="$1" shift 1 printf "%s\n" "$TEXT" | grep -q "$@" }</code></pre> Example usage: matches "$FOO" "^[0-9]$" && echo "FOO is a number! :)" || FOO=0</code> It can also be useful to implement some kind of heuristics based on which patterns occur in a text. Bash Arrays</h2> Bash has arrays so one doesn't have to care about delimiters too much. Note on compatibility: These arrays only work in bash which isn't isntalled everywhere (dash is a popular alternative). Please use #!/bin/bash</code> instead of #!/bin/sh</code></s> to indicate that your script requires bash to work. Create an array: empty_array=() array=('text' "can go" "$( echo here )" ";")</code></pre> Appending to an exisiting Array: array+=("foo") array+=("add" "multiple" "elements")</code></pre> These can then be accessed with 0 based indices: echo "${array[1]}" # prints out: can go</code></pre> To get the number of elements in an array: echo "${#array[]}" echo "${#array[@]}"</code></pre> An array can expand to multiple arguments: printf "-> %s" "${array[@]}" # Compare with the output of: printf "-> %s" foo bar baz</code></pre> Which is also useful for for</code>-loops: for i in "${array[@]}" ; do echo "-> $i" done</code></pre> And to clean up: # remove element at index 1 uset array[1] # remove the whole array uset array</code></pre> There is more to bash arrays (including associative arrays) … Check out The Bash Reference Manual on Arrays </a> See Also</h2> ANSI Escape Sequences</a></li> Portable Shellscripting</a></li> </ul> Having a look at greetd 2022-08-16T00:00:00+00:00 What is greetd?</h2> greetd is a minimal and flexible login manager daemon that makes no assumptions about what you want to launch. </blockquote> In other words Kenny Levinsen</a> created an awesome piece of software that allows anyone to easily launch anything on login and that without being bound to a fronted. You can find greetd on sourcehut </a> Most information in this article came from reading the greetd code and making notes on good old paper. Note: Kenny Levinsen also wrote an interesting blogpost on login managers.</a> Note: If you want to understand how greetd works please read the code yourself, this article alone is not enough! (but hopefully helpful) In case you find a mistake or have feedback please contact me.</a> How greetd is structured</h2> Greetd has a pretty modular approach to things and is made of a bunch of components one can look at separately. The managing greetd server daemon (main focus)</li> The greetd --session-worker</code> process (good to know)</li> The greeter</li> </ul> I won't be looking at the greeter as the interface is pretty well described in the manual</a>. The other two are both compiled into the same binary, the main function decides which component it should become after parsing the command line parameters. The common part</h2> There is a small common part that always happens when the greetd binary starts up: It invokes the commandline and configuration parser to get its complete configuration.</li> It invokes mlockall()</a></code> to prevent sensitive data like buffers containing passwords from being swapped to disk.</li> Then it decides what to become.</li> </ol> The greetd session worker</h2> The simpler part of greetd is the actual session worker, so I'll start with that … The relevant worker()</code> function in greetd/src/session/worker.rs</code> </a> The session worker is started by starting greetd with the --session-worker</code> or simply -w</code> flag and passing the number of a file descriptor that is supposed to be a Unix socket for exchanging commands and feedback. What it does</h3> The session worker itself is one strand of pretty straightforward code that feels a bit like a shellscript that asks for input and aborts with an error message on failure. It does a few things: Wait for an InitiateLogin</code> or Cancel</code> message.</li> Setup a conversation with PAM</abbr>.</li> Let a PAM adaptor implemented in a separate file</a> do the authentication conversation.</li> Do pam_acct_mgmt()</a></code> and pam_setcred()</a></code> and send a Success</code> message.</li> Wait for a Args</code> (sets the command for ) or Cancel</code> message and answer Success</code>.</li> Wait for a Start</code> or Cancel</code> message.</li> Fetch user information from PAM.</li> Set a session id using setsid()</a></code> to make the process the root of a new session.</li> If we have a specific virtual-terminal configured. Tell PAM that we want to run in that tty.</li> Set the XDG_VTNR</code> environment variable via PAM.</li> Open the terminal file, set the tty to text mode, clear it and switch to it if necessary.</li> Use the greetd terminal abstraction to do the plumbing for the pipes</a> and switch the controlling tty</a>.</li> </ol> </li> Set the pwd</abbr> for the new session, either to the users home or the filesystem root as fallback.</li> Set a whole bunch of environment variables for the session (including the GREETD_SOCK</code> for the greeter, and the ones received with the Args</code> message) using PAM.</li> Prepare the command to execute in the new session (shell + profile files + session command)</li> Fetch the environment variables for the new session from PAM.</li> Fork</li> </ol> After the fork the parent sends the pid</abbr> to the the greetd server process and closes the socket, sets its parent death signal</a> to SIGTERM</code> and waits for the child thread to terminate before ending the PAM session. The child process drops its privileges, also sets its parent death signal</a> and then uses execve()</a></code> to launch the prepared command with the environment provided by PAM. Note on Environment variables: The variables received using the Args</code> call get set before the ones coming from the greeter to avoid overriding some important variables. This behaviour was introduced</a> after making it possible for the greeter to set the environment variables (again)</a>. Personal opinion: If you have a good reason for changing those you shouldn't be doing that through the greeter anyway. Messages between Session Worker and Server</h2> For later reference here are the messages that are exchanged between session worker and server via given socket using a JSON based protocol. Parent to Session Messages</h3> InitiateLogin</code></h4> Message sent to start the login process. string service </dt> The PAM service id so that PAM knows which configuration to apply </dd> string class </dt> ends up in the XDG_SESSION_CLASS</code>, either 'user' or 'greeter' </dd> string user </dt> the username to log in as </dd> bool authenticate </dt> set to false</code> if we have an autologin session </dd> TerminalMode tty </dt> The description of the terminal to attach to </dd> bool source_profile </dt> Whether or not to read the profile files before executing the session command </dd> </dl> PamResponse</code></h4> A response to a question from PAM. string? response </dt> The response, may be null </dd> </dl> Args</code></h4> Sets the command to be executed for the session when Start</code> is sent. string[] cmd </dt> An array of arguments representing the command </dd> string[] env </dt> An array of environment variables to pass to the given command (it was added (back) on 2022-08-13</a>) </dd> </dl> Start</code></h4> Gives the final go after the session is full set up. Cancel</code></h4> Can be used at almost any step to cancel the session setup. Session to Parent Messages</h3> Success</code></h4> What it says on the tin. Signals that a command has successfully finished. Error</code></h4> Signals that an error has been caught somewhere. Error error </dt> A rusty error response, probably gets serialised to an error message </dd> </dl> PamMessage</code></h4> A Question or other Output from PAM. AuthMsgType style </dt> an enum telling what kind of message this is and what the expected response is, one of Visible, Secret, Info or Error </dd> string msg </dt> a string intended for the human trying to log in </dd> </dl> FinalChildPid</code></h4> Sends back the process id of the session child after the session was launched. uint pid </dt> the process id </dd> </dl> The greetd server</h2> The server process is the one started by the init-system, it is responsible for launching the session workers and telling them what to do, it also houses the internal state machine that knows which session to launch next, it also does some "managing" for the processes in the session. The server is made up of multiple components: A base to orchestrate the startup and running sequence, also handles signals</li> A Context</code> and and ContextInner</code> to keep track of multiple sessions and scheduling</li> Session</code> objects that launch the session workers and handle the communication</li> A function that does the communicating with the greeter</li> </ul> What it does …</h3> … on startup</h4> The entry point is the main()</code> function in the greetd/src/server.rs</code> file </a> After the initialisation has decided that it will be a greetd server the main function also does a few things: Determine the service name it will tell PAM based on which configurations for PAM are available. (warns when using the login</code> fallback)</li> Get some info about the user configured for the default_session</code></li> Create the Unix Socket (Listener</code>) for the greeter, set permissions and set the GREETD_SOCK</code> environment variable.</li> Get information about the tty it is supposed to be on. Returns an error told to wait for the terminal to become active it waits (terminal.switch</code> set to false</code> in config).</li> Create a Context</code> to do the housekeeping for us.</li> Start initial session or greeter for the default session depending on which one is configured. (Other state transitions after initial bootstrapping are handled by the Context</code>).</li> Create the runfile to know the initial run apart form the others in terms of system restarts, not greetd restarts.</li> Loop to handle posix signals (SIGALRM</code>, SIGCHLD</code>, SIGTERM</code> and SIGINT</code>) as well as Incoming connections on the greeter socket.</li> </ol> … on an incoming connection from the greeter</h4> The connections are accepted using the main-loop (see previous section) the handler gets the bidirectional datagram stream and a copy of the Context</code> structure (the inner context staying the same, this is how the information about the current global state is exchanged between possibly multiple connections). The client_handler()</code> in greetd/src/server.rs</code> handles the communication </a> Inside this function is a loop that reads the request and then decides which function to call on the Context</code>, wraps up the response and sends it back. CreateSession </dt> Calls the create_session()</code> function with the given username, if successful, responds with the first question </dd> PostAuthMessageResponse </dt> Calls the post_response()</code> function to answer the PAM question, fetches and returns the next question or the Ready</code> response if that didn't fail </dd> StartSession </dt> Calls the start()</code> function and returns the result </dd> CancelSession </dt> Calls cancel()</code> function and returns the result </dd> </dl> There are two helper functions: client_get_question(context) </dt> This function fetches the next question from the context using the get_question()</code> method, if it returns a question it packs that into an object that can be serialised and sent back, if not it calls wrap_result()</code> on the result and returns that, meaning automatic error handling or an automatic success message in case of an ok. </dd> wrap_result(result) </dt> This function takes a Result</a> and spits out a spendable success or error message containing appropriate information about the error. </dd> </dl> Context</h2> The Context</code> mechanism is implemented in greetd/src/context.rs</code> </a> As already stated greetd manages its state using Context</code> objects (no idea what the correct rust terminology is, but its a data-structure with a set of functions and quacks like an object). To be more precise it uses two context objects, the outer Context</code> for configuration and connection state and the ContextInner</code> for storing global state of a 3-stage session pipeline that is shared between all interested parts of greetd. Datastructures</h3> The Context</code> object holds the following values: RwLock</a><ContextInner> inner </dt> The ContextInner</code> object for global state wrapped in a lock that allows shared reading or exclusive writing </dd> string greeter_bin </dt> The command for the greeter </dd> string greeter_user </dt> The user the greeter runs as </dd> string greeter_service </dt> The PAM service name for starting greeter sessions </dd> string pam_service </dt> The PAM service name for starting normal sessions </dd> TerminalMode term_mode </dt> The vt in which this context spawns sessions </dd> bool source_profile </dt> The source_profile</code> setting </dd> string runfile </dt> The path of the runfile for persisting sate across possible greetd restarts </dd> </dl> The ContextInner</code> has the following fields: SessionChildSet? current </dt> Information about the currently running session </dd> SessionSet? scheduled </dt> The session that will be started once the currently running session terminates </dd> SessionSet? configuring </dt> Stage before scheduled</code>, a session that isn't ready to be launched just yet because the greeter is still answering the questions PAM has </dd> </dl> The SessionSet</code> here is a Session</code> (representing a session that is being set up) with a timestamp attached. The SessionChildSet</code> stores a SessionChild</code> (representing a running session) and has an is_greeter</code> flag in addition to the timestamp. Note: There is a distinction between the Session</code> and SessionChild</code> object because a session that is already running doesn't care about all of the communications foo and is only interested in finding out if the session is still running and ending it if it isn't supposed to be running anymore, the only shared value is the pid of the session worker. Methods</h3> The methods on the context seem to be made to make it easy to write down what should happen from the outside, also they are pretty well commented so you know how they are supposes to be hooked up. start_unauthenticated_session()</code></h4> Directly start an unauthenticated session, bypassing the normal scheduling. </blockquote> This one translates to spawning a session worker and sending it all commands that should be needed for an autologin session and gives empty answers for any questions PAM asks. This is used for the initial_session</code> and the sessions for the greeters. It returns the created Session</code> as an object. Username, session-class, PAM-service and the command are given as arguments. start_greeter()</code></h4> Directly start a greeter session, bypassing the normal scheduling. </blockquote> A small convenience wrapper that preconfigures the start_unauthenticated_session()</code> with stored settings for a greeter session, uses the greeter_service</code> as the PAM service. It also returns the created Session</code> object. greet()</code></h4> Directly start a greeter session, bypassing the normal scheduling. </blockquote> This one starts a greeter session and sets it as the current</code> session in the inner context. It returns an error if that current</code> session is already occupied. start_user_session()</code></h4> Directly start an initial session, bypassing the normal scheduling. </blockquote> This will use start_unauthenticated_session()</code> to autologin a given user with the given command. (used to implement the initial_session</code>) create_session()</code></h4> Create a new session for configuration. </blockquote> Creates a session preconfigured with the Context settings and the given username and swaps it into configuring</code>. Also mitigates a race-condition by cancelling a swapped out session. Returns an error if there is no current</code> session or there already is a session configuring</code> or scheduled</code>. cancel()</code></h4> Cancel the session being configured. </blockquote> get_question()</code></h4> Retrieve a question from the session under configuration. </blockquote> Uses the get_state()</code> method on the Session</code> and returns either an ok when there are no more questions (Ready</code>) or the question that is currently open. post_response()</code></h4> Answer a question to the session under configuration. </blockquote> Calls the post_response()</code> on the currently configuring</code> Session</code>. start()</code></h4> Schedule the session under configuration with the provided arguments. </blockquote> Returns an error if the configuring</code> session has not signalled that it is ready yet or isn't present. Sets the sessions command to the given value, and swaps it into the scheduled</code> stage, a session that already was in scheduled</code> will be cancelled. Sets a timer to fire the SIGALRM</code> signal in five seconds which will be received using the alarm()</code> method. alarm()</code></h4> Gets called by the main-loop when a SIGALRM</code> (timer) arrives. Is a noop if no session is scheduled. If the current session is still running it will be sent a SIGTERM</code> or SIGKILL</code>, depending on how much time has elapsed since the session was scheduled. The SIGALRM</code> will be set up to fire again in one second. If the current session is cleared the seduled</code> session will be sent a Start</code> command and be turned into the current</code> session. check_children()</code></h4> Gets called by the main-loop when a SIGCHLD</code> (a child process has terminated, usually) arrives. This function calls the waitpid()</a></code> function with the NoHang</code> flag set to tell it that we don't want to wait if there is nothing new it can tell us. It can react to a bunch of events: In case of a StillAlive</code> or an error indicating that we currently don't have any children the function exits. </li> States indicating some kind of normal event but not matching as well as Interruptions get thrown away. </li> Something exited or was killed with a signal and the current</code> session owns that pid: If a session is scheduled</code> we try to start it and make it the current</code> session, if not and the session was our greeter return an error, in case of a normal session use start_greeter()</code> directly and make it the current</code> session. </li> </ul> terminate()</code></h4> Notify the Context that we want to terminate. This should be called on SIGTERM. </blockquote> Shuts down the configuring</code>, scheduled</code> and current</code> session while holding the write lock to avoid race conditions and sessions being rescheduled. State Machine</h3> All these functions form the state machine that cycles between greeter and user sessions and makes sure the pipeline from configuring</code> to current</code> doesn't get clogged up by anything misbehaving and that greetd cleans up after itself. The Session Interface</h2> The session interface is implemented in greetd/src/session/interface.rs</code> </a> To make the external session processes and the context talk to each other Greetd has the Session</code> class. It wraps an entire session process and handles the communication. It also remembers the last incoming message as last_msg</code> and the session processes pid</code>. However, after setting up the session (as noted earlier) the Communication channel becomes useless as we now only care about the sessions current running status detecting when it exits and ending it if it misbehaves. This is the reason a second object, the SessionChild</code> exists which holds the session managers pid as task</code> and the pid of the process in the session as sub_task</code>. Methods of Session</code></h3> new_external()</code></h4> Creates the Session</code> object and spawns a session worker process with a datagram socket attached for communication. initiate()</code></h4> Send an InitiateLogin</code> message to the session worker. get_state()</code></h4> Tries to receive a message from the session worker if there is no last_msg</code> and stores it in last_msg</code> Then it returns a SessionState</code> struct contain either a question from PAM or a Ready</code> if PAM has no further questions. This is used in the Context.get_question()</code> method. cancel()</code></h4> Sends a Cancel</code> message and resets last_msg</code>. post_response()</code></h4> Send a response to an authentication question, or None to cancel the authentication attempt. </blockquote> Sends a PostAuthMessageResponse</code> message and resets last_msg</code>. send_args()</code></h4> Send the arguments that will be used to start the session. </blockquote> Since 2022-08-13</a> this also sends a list of environment variables to match the new Args</code> message. Also reads a message after that and returns whether the command was successful. start()</code></h4> Sends a Start</code> request and waits either for an Error</code> or a FinalChildPid</code> answer while dismissing questions from PAM with empty answers. After that it shouts down the socket and returns a SessionChild</code> object as the Session</code> object mainly made for session setup is no longer useful once the session is running. Methods of SessionChild</code></h3> owns_pid()</code></h4> Tests if the task</code> equals the given process id. (Used for checking if the session terminated on a SIGCHLD</code> signal in the Context</code>) term()</code></h4> Used to send the session process (sub_task</code>) a SIGTERM</code> to tell it that we want it shut down. kill()</code></h4> Sends a SIGKILL</code> to both, the session manager and the session process. Used by the Context.alert()</code> method when a session takes too long to shut down. Updates to this Page</h2> 2022-09-11</h3> Added information about the commits adding back the environment variables and another round of spellchecking. May also have broken some links (I hope not too many). 2023-03-03</h3> Added Link to Kenny Levinsens blogpost. But Why?</h2> The reason I created this writeup is to first of all have an understanding of what my login manager of choice does and also to have a reference when tinkering with it. One of my plans with this includes making greetd capable of running multiple sessions simultaneously to allow user switching and to be able to reuse the greeter for the initial login for locking my screen. In case you have found a mistake, have questions or just feedback in general … … please contact me! </a> Building a crappy cheap LightDM 1/? 2022-08-05T00:00:00+00:00 This is a lightly edited mind dump, don't expect a well written post here. What it is, does and why I built it</h2> The goal of this little project was to build a collection of tools that is able to switch between two sessions on a Linux system for the purpose of having a session for working and one for logging in and unlocking the screen, the way it is implemented is by changing virtual terminals</a> to the corresponding session when certain state-changes occur, much like LightDM</a> and the light-locker. The reason I'm doing this is because I want to be able to replace my currently pretty barebones screenlockers with a rich screenlocker that is able to do more than just showing me an unlock form, mostly for my phone but also for my other machines. Why not just use LightDM?</h3> Maybe oversimplified answer: I have the opinion that LightDM isn't great on mobile. Implementation</h2> The core of this contraption are two instances of greetd</a>, one for actually logging in and one for giving me a properly set up session to run my compositor in which could be replaced by an instance of tinydm on my phone. These two communicate by always running one of two scripts the locker and the unlocker, both grab the current vt with the chvt</code> command, write to pidfiles and kill</code> each other and run hooks on exit. Please Note: For my current implementation the user that runs the logged in session needs passwordless sudo access to the chvt</code> command, in a multiuser scenario this may not be the best idea. Unlocking</h3> Unlocking works by entering your credentials in the greeter, triggering the unlock</code> script which sends a SIGUSR1</code> to the locker process (using the pidfile), and then entering a sleep loop waiting for the next session lock. The lock</code> script handles this SIGUSR1</code> by killing the lockscreen (in my case swaylock) and using chvt</code> to switch the currently active terminal to the one of the real-work-session. Locking</h3> Locking works almost exactly the reverse way as unlocking. The lock</code> script starts the screenlocker, kills the unlocker, this time with an ordinary SIGKILL</code> and starts waiting for being unlocked. The unlocker</code> uses chvt</code> to grab the vt with the greeter on it and exits telling greetd to start the greeter session for the next unlock. Bootstrapping</h3> To solve our little problem of how to bootstrap this thing there is also an initial-session-lock</code>, a script that writes its pid to the lockers pidfile and only exits with a success code when it receives a SIGUSR1</code> to act as a trigger for starting the session on the first unlock. For shutting down my current solution is to wrap this in a loop and kill the unlocker after sway exits. Code</h2> Currently this is an experiment I've run on my personal dotfiles so the sources are not really in any central location yet and you probably don't want to just copypasta my scripts. Scripts</h3> My personal screen-lock</code> script</a></li> My unlocker</code> script</a></li> My initial-session-lock</code> script</a></li> </ul> Greetd configuration</h3> As mentioned above I have two instances of greetd running, one that starts the unlocker on login and one that starts the actual session. A pretty standard greetd configuration with the unlocker</code> script instead of a real session.</figcaption> [terminal] # The VT to run the greeter on. Can be "next", "current" or a number # designating the VT. vt = 7 # The default session, also known as the greeter. [default_session] # Starting tuigreet here which starts the unlocker command directly to unlock the second session command = "tuigreet --cmd unlocker -r -t" # The user to run the command as. The privileges this user must have depends # on the greeter. A graphical greeter may for example require the user to be # in the `video` group. user = "_greeter"</code></pre> </figure> The one starting the actual session could be replaced by something like tinydm</code> as I only care about the autostarting part and that the environment is properly set up so I don't have to do it myself here. A greetd configuration that uses the `initial_session` mechanism to autostart a shell oneliner that starts our session.</figcaption> [terminal] # The VT to run the greeter on. Can be "next", "current" or a number # designating the VT. vt = 8 [general] # Don't know if actually necessary but I don't want to restart the session I'm writing this post in runfile = "/run/slatian-greetd-initial-session" [initial_session] # The command that wits for the initial unlock, if successful starts the session and returns the control to the lockscreen after the session ends before it starts over. command = "sh -c 'while true; do initial-session-lock && dbus-run-session sway; kill \"$(head -n 1 \"$XDG_RUNTIME_DIR/unlocker.pid\")\"; done'" user = "slatian" # default session omitted … nothing interesting there</code></pre> </figure> Sudo configuration</h3> The following snippet placed in a file in /etc/sudoers.d/</code> will give everyone in the wheel</code> group (the group that has sudo access anyways, may also be called sudo</code> on some systems) the permission to run chvt</code> without having to enter a password. %wheel ALL= NOPASSWD: /usr/bin/chvt</code></pre>Almost EOF …</h2> More or less obviously this is very hacky, has multiple issues and falls apart very easily, somewhere in the next few months I'll try to find a cleaner way to implement this without relying on too much on shell scripting and firing signals at processes using pidfiles. Until that happens I'll continue to use my sxmo screenlocker</a> on my phone because it mostly works well enough for what I need. Screwing around with Sliderland 2022-06-14T00:00:00+00:00 Overview</h2> Sliderland</a> is a coding playground made by blinry</a> that evaluates some javascript and displays the result using 64 sliders (the original used 64 html sliders, but they were replaced in favour of a canvas that draws things that look like sliders because of browser compatibility). I also host a Sliderland on sliderland.slatecave.net that has a dark mode and behaves more like a scrollable page.</del> Some things I made with Sliderland follow, feel free to tweak them have fun and share them! (mentioning my name somewhere would be nice but isn't required) Screw it! Demo</h2> The "final" Screw demo that cycles through a few wavy things that look a bit like screws. </a> A barebones screw that is a bit easier to understand. </a> Roofs and Birds</h2> A scrolling wave of bars with roofs on them. </a> Same demo as above with a glitch effect switching between bird and roof mode. </a> Tutorial: Smart ssh jumping 2022-06-05T00:00:00+00:00 I'm assuming you know what an ssh jump host is, why you want it and how you configure it in your ssh ssh_config</code> file. The problem I'm trying to solve</h2> SSH jumping (via JumpHost</code>) is usually useful when you have a bastion host you have to log into to reach the network behind. In some environments you never reach this "private" network in any other way, but if the private network belongs to your home- or office and the bastion is for connections from the big scary intertubes you want to use it only when necessary, meaning we need some kind of conditional ProxyJump</code>. Example scenario</h2> Lets assume we have the following two hosts: tesla </dt> This is our bastion host publicly reachable </dd> hawking </dt> This is our private host we actually want to connect to </dd> </dl> The easy workaround</h2> The easy solution would be to have multiple aliases and using your brain to make the decision by using the hostname when at home and the hostname-remote when elsewhere. Example ssh configuration with different aliases for local and remote connections:</figcaption> Host tesla HostName ssh.example.org User nicola IdentityFile ~/.ssh/id_foo Port 22222 Host hawking User stephen IdentityFile […] Host hawking-remote ProxyJump tesla User stephen IdentityFile […] HostName hawking Port 22</code></pre> </figure> Problem is that you have to type the remote alias every time you're not at home. Also using git this way is pretty annoying (you have to set up multiple remotes and manually tell git which one to use). Smart solution</h2> The smart solution would be that ssh somehow could figure out if it is "at home" and if not enable the proxy jump option. For that we have to answer the following two questions: Are we at home?</li> How do we tell ssh to (not) use the ProxyJump?</li> </ul> Are we at home?</h2> To find out if we are at home there are several options: Querying DNS and for some address that is different in our home network - Too complicated to set up and is a PITA</abbr> with slow resolvers (ssh connections already take 3 seconds to establish on school WiFi 🙄)</li> Comparing the WiFi name - This is not an option for wired networks.</li> Manually - forget it I wanted to get rid of that</li> </ul> All of the above are not really desirable, so lets think of something else … One thing that is usually frowned upon is fingerprinting, however if we turn the concept around that your local machine fingerprints its environment all of the sudden it becomes a pretty good tool. If the year is somewhere after 2020 you probably have IPv6 configured and if you are someone like me, you probably have your own local prefix which is a pretty unique identifier for a network which the router is nice enough to tell your device when it connects! So the magic for finding out if you are at home becomes an ip a</code> piped into a grep -F "$ip_address"</code>. For my purposes I wrote a little script called has-ip-grep</code> that is a simple shell pipe returning success if a given "fingerprint" matches and fails if it does not. The has-ip-grep</code> script which tests if the first argument given to it matches the IP-address output part of an ip a</code> command.</figcaption> #!/bin/sh set -e ip a | awk '/inet/ { print $2 }' | grep -F "$1"</code></pre> </figure> Explanation: ip a</code> prefixes the addresses assigned to interfaces with inet</code> or inet6</code> the awk command therefore looks for lines containing the string inet</code> and if it finds one outputs the second column which is the IP-address. The output gets piped into a silent grep command that is told to look for instances of the first script argument if that's not the case. The set -e</code> makes sure the script returns an error code if that match fails. Examples</h3> In the first example we try to match against our local IPv4 loopback interface which should always return a success, which works great for a quick test! has-ip-grep 127.0.0</code></pre> Example 1: Succeeds because we have the IPv4-loopback-address assigned to our lo</code> interface.</figcaption> </figure> The second example show matching against the prefix usually used by a popular series of home routers made by a German company, this is also great for testing but it will very likely not be enough to identify your home network because your neighbours and your friend's network probably have the same fingerprint. has-ip-grep 192.168.178</code></pre> Example 2: Usually succeeds if you are connected to a popular series of German home routers.</figcaption> </figure> Other problems are that because we match substrings here this will only be convenient to match against ranges that align nicely with the decimal notation and that we get false positives, for example if someone assigns us a 10.192.168.178</code>. We can reduce the probability of false positives by matching against our IPv6-address, replace the fd45:6789:abcd:</code> in the third Example with your own local IPv6-prefix. I don't recommend you rely your global unicast prefix as it is usually assigned by your ISP</abbr> and may change behind your back. has-ip-grep fd45:6789:abcd:</code></pre> Example 3: Will succeed in your home network and maybe some other random network half around the globe you will never know exists.</figcaption> </figure> So now we have a pretty good (works most of the time) way to know if we are at home lets put it to use! Conditional ProxyJump</h2> Luckily ssh has exactly what we need to combine it with our command we just created. You are probably familiar with the Host</code> keyword in your ssh configuration, turns out if you have a decently modern version of ssh there is also a Match</code> keyword which supports some arguments. See the manual: man 5 ssh_config</code> </a> With that knowledge you can match against the target host (if you don't you'll run into funny recursion …), the network fingerprint of your choice and arrive at a configuration that can decide on the proxy without having to query your brain. Example ssh configuration which automagically knows how to connect.</figcaption> Host tesla HostName ssh.example.org User nicola IdentityFile ~/.ssh/id_foo Port 22222 Host hawking User stephen IdentityFile […] Match Host hawking !Exec "has-ip-grep '192.168.42' && has-ip-grep 'fd42:f00b:aba2:'" ProxyJump tesla HostName hawking Port 22</code></pre> </figure> Explanation</h3> The configuration for tesla stays the same (assuming we only want to connect to it from the public side). The configuration for the local connection to hawking can stay too, the Host hawking-remote</code> directive was replaced with a Match</code> that tests if we want to connect to hawking and then executes our shell command. In this case we check for two IP-addresses that we expect to get assigned in our private network, As I'm running dual stack it won't hurt to check against the IPv4-address too. These two commands are combined with some shell syntax and quoted using double-quotes because that's how ssh_config</code> works. The Exec is prefixed with an exclamation mark because we want to match if we are NOT at home. The part after that is the same as with the remote alias except for the User and IdentityFile because we now the Host hawking</code> always matches and we don't have to duplicate them anymore. I can imagine it being a lot of fun configuring several Networks this way 🙃. Security</h2> This is by no means any kind of security measure, if the network you are connected to matches the fingerprint of your private network this will make the wrong decision to connect directly! However if you are running such a setup you should have the certificate fingerprints of your real servers already in your known_hosts file and then just be smart enough to not accept any new certificates over untrusted connections. If that finger print matches when you are not in your network (assuming you randomly chose an IPv6-prefix) you should get the hell out of there! Debugging</h2> ssh -v</code> is your friend! If ssh seems to do nothing except burning CPU cycles you might have ran into some recursion problem, Make sure that your Match</code> keywords always match on some host and that your proxy connections aren't jumping in circles! Also some pits I fell into (all of them are documented in the ssh_config</code> man-page): If your forget the Host</code> rule on a Match</code> you'll get some recursion going.</li> A Host</code> rule ends when a Match</code> rule starts.</li> When you assign a variable multiple times only the value from the first assignment is taken (especially important when you want to rewrite the host for short aliases and later to localhost when redirecting into a tunnel which won't work).</li> </ul> That's it!</h2> Have fun connecting to your servers without having to think about changing names! If you spotted a mistake somewhere or have other feedback feel free to contact me! Integrating swaylock-mobile with Sxmo 2/? 2022-05-07T00:00:00+00:00 Sxmo 1.9.0 was released last weekend</a> for PostmarketOS edge, which means … I have continued my work on the screenlocker as promised. In this part I'll mostly explain what needed to be done (and why) for the screenlocker to work. Related Resources</h2> Part 1 documenting how the screenlocking part of sxmo works under the hood</a></li> Screenlocker Integration Dotfiles</a></li> Swaylock mobile, my screenlocker</a></li> </ul> What my current solution can and can't do</h2> Current sate of swaylock-mobile is, that it's a pretty standard swaylock that fits on a touchscreen (even when rotated!) with a keypad for entering numeric pins without a real keyboard, that is enough to integrate it into Sxmo and get some additional security for my phone in daily use. So the integration currently handles: Locking the screen, before turning it off</li> Proximity-lock for both locked and unlocked states</li> State-management for the idle locker and lisgd</li> </ul> More important, what does it NOT handle: X integration, I have not tested it with the dwm version</li> Graceful degradation, fallback in case there is no screenlocker doesn't work</li> Maybe I have broken some suspend inhibitors … I did</li> Remaining slightly broken states</li> </ul> Changes</h2> New Wrapper script run-screenlocker</h3> This is a script that runs swaylock-mobile until it exits with a success status-code (the warning that you should make sure it is restarted in case of a crash still applies) and after that calls the unlock hook. This script gets called from the lock hook and should be the only thing that can call the unlock hook. (one exception would be the fallback …) Hooks that were modified</h3> sxmo_hook_inputhandler.sh </dt> The Powerbutton now only toggles screenon and screenoff and calls the lock hook before off, also the corner-swipe locks the screen and turns it off. </dd> sxmo_hook_lock.sh </dt> Just does, what it says on the tin: locking, also makes sure it only locks when called from an unlocked sate, also invokes the new idlelocker hook. </dd> sxmo_hook_proximitylock.sh </dt> The proximitylock now calls the screenon hook instead of the unlock hook, also disabled the can_suspend locks. </dd> sxmo_hook_screenoff.sh </dt> Now uses $SXMO_STATE.screen for storing screen state. </dd> sxmo_hook_unlock.sh </dt> Now uses the idlelocker hook for setting the screenlocker. </dd> </dl> Hooks that were added</h3> sxmo_hook_idlelocker.sh </dt> Sets the idlelocker for screenon scenarios, depending on the SXMO_STATE. </dd> sxmo_hook_screenon.sh </dt> Gets called for turning the screen on (and some screen related services). </dd> </dl> New state-file</h3> There is a new file to keep track of things, previously $SXMO_STATE was used for storing three states sxmo could be in: screenoff, locked and unlocked. These states would still be sufficient, but the model with a state for the lock (locked and unlocked) and a state for the screen (on and off), which allows me to simplify the previously error prone and complex state machine into two state-machines which just turn things on and off without having to make assumptions about the screen-state or the lock status. For that I needed a second state-file though which I called $SXMO_STATE.screen for now (clean solution would be to introduce a $SXMO_SCREEN_STATE and $SXMO_LOCK_STATE). New environment variable: SXMO_REASON</h3> Before rewriting the state-machines, I tried to work with the old states and figuring out what to do depending on where the call came from, which turned out to be a horrible buggy mess, but its the reason SXMO_REASON was introduced. Why is it still there you ask: Because if the screen is locked and the proximity-lock triggers in most of the cases you don't want the screen to turn on again by just uncovering the proximity sensor… the screenon hook script currently uses it to cancel if this variable is set to proximity-lock and the lockscreen is active. (Someone could make that a preference, however it won't work when the system goes to sleep) Additional sway configuration</h3> One thing that isn't immediately obvious is that you want to turn your screen back on again and that it doesn't work with the default configuration, the reason being that sway detects when a proper screenlocker is active it only allows keyboard bindings (the power-button behaves as if it was the XF86PowerOff key on a keyboard) when they are bound with a --locked switch, so I put a bindsym into my sway config file, the cleaner way would have been to update sxmo_swayinitconf.sh (I did it that way because I hope this way it will break less often than modifying the swayinitconf script). Fixing the remaining TODOs</h2> It doesn't work with X</h3> I (or someone who knows sxmo) would have to make sure all modifications are also applied to the dwm version (not sure about how to solve the power-button problem though) Graceful degradation in case of missing/disabled screenlocker</h3> My approach would be a dummy screenlocker, that creates a file somewhere and then watches it for close_writes with inotifywait (and deletes it after), the handler for the power-button in the file exists + locked + screen on would touch that file, which would trigger an unlock. Broken suspend inhibitors</h3> The proximity-lock hook by default locks the can_suspend mutex, I leave it on for the lock-screen which blocks the suspend script from doing its job, that's not battery friendly … Lets just bypass that by disabling the lock and hope it won't break. Slightly broken states</h3> There are two scenarios I know of where the state handling is slightly broken, both involve the proximity-lock: When unlocked + proximity-lock active (sensor covered) + power button the screen turns on for a second and then goes to lock + screenoff. I currently don't know why exactly this happens as the power-button should check against the screen-state and toggle it once, turning it on …</li> When locked + proximity-lock active + power button the screen turns on (like I want it when unlocked, ironically) and stays on for the 8 seconds timeout when not touched, this can be explained by how the proximity-lock currently triggers on changes (which is very good, because it saves power!)</li> </ul> Integrating swaylock-mobile with Sxmo 1/? 2022-03-18T00:00:00+00:00 I have been trying to introduce my touchscreen based screenlocker to Sxmo for over a week now. Turns out it is not as easy as I thought. In this part of the sersies I'll try to describe what sxmo does for screenlocking and what changes with the 1.9.0 release. The actual implementation will be discussed in part 2. What I have tried so far</h2> My first approach was to try out the desktop screensaver, the problem was, that it tried to interact with the newer version of Sxmo, so I reverted to messing with the default screenlock hook. This at first seemed simple, just insert the screenlocker after the service starting and stopping foo, lock my phone, the screenlocker comes up as expected, I try to enter my pin and … nothing, I forgot that Sxmo turns off the touch input for the locked mode, so I'm not able to enter my pin on the touchscreen. Ok easy fix turn the touch back on after starting the screenlocker … and yes I'm successfully able to unlock my phone, but for some reason the screenlocker comes up very soon and after coming Up my phone won't go to sleep … The reason turns out to be that the sxmo_screenlock.sh script figures out the current state by looking at what it turned off. That means, I broke that by messing with the touchscreen state, which brings us to the present. Where I'm now</h2> This is the "legacy" script that still runs on my phone which will be removed when Sxmo updates to 1.9.0. See the next section for the new solution. This script seems to be the pint where everything related to locking and sleeping goes through. Setting state</h3> It defines the following functions for setting state: lock </dt> Turns touch input off </dd> unlock </dt> Turns both touch input and the screen back on </dd> off </dt> Turns the screen off </dd> crust </dt> Sends the phone to deep sleep and determines why it woke up </dd> </dl> I probably want to to add some semantics here to find out why the locking/screenoff happened and launch my screenlocker if the reason is an idle session or a power button event. These functions get called from a few places: core/sxmo_inputhandler.sh </dt> Has a lock_screen_action() function for when the screen is locked to tun it off, into locked and back into unlocked. </dd> Also calls the lock function when you swipe away from the bottom left corner. </dd> core/sxmo_screenlock_deeper.sh </dt> Similar to the the lock_screen_action() above but is intended for being called after a few seconds of idle time and it won't move to an unlocked state. </dd> core/sxmo_proximitylock.sh </dt> This script periodically checks if the distance sensor is covered and then makes the decision to call the unlock or the off function, also based on the state of the screenlocker. </dd> core/sxmo_rtcwake.sh </dt> This script seems to be called by nothing … it calls the crust function on sxmo_screenlock.sh </dd> default_hooks/contextmenu </dt> Calls lock, off and crust with the intention of explicitly locking the screen. </dd> notifications/sxmo_notificationwrite.sh </dt> Calls the lock function if the screen is off, probably with the intention to make the screen light up when there is a notification. </dd> </dl> Idea from wiring that: start the screenlocker as a sxmo_service from a blocking-hook and make the unlock function refuse to work as long as it is running, execute the unlock when it returns, if no lockscreen hook is specified, fallback to turning off the touchscreen. Querying state</h3> These states from above can also be queried with the getCurState function, it returns: unlock </dt> When unlocked, this is returned if the touchscreen is on </dd> lock </dt> When locked with the screen on, this is returned if touch is off but the screen is on </dd> off </dt> When locked with screen off, this is returned when both touch and screen are off </dd> </dl> This will need some modifications as I'm breaking the assumption that touch state equals locked state. Quite a few scripts rely on this one: notifications/sxmo_notificationwrite.sh </dt> Tests for the off state to turn he screen on to get attention. </dd> core/sxmo_inputhandler.sh </dt> Uses this to cycle states. </dd> core/sxmo_screenlock_deeper.sh </dt> Uses this to determine the next deeper lock/sleep state. </dd> core/sxmo_proximitylock.sh </dt> (Ab)uses this to avoid unnecessary calls to unlock and off. But is not really context aware. It probably gets stopped elsewhere. </dd> core/sxmo_rtcwake.sh </dt> Cronjob, Only goes to crust if not in an unlocked state. (It has other factor too, but not being unlocked is a requirement) </dd> </dl> Looking at this I'm not really willing to break this function, maybe some kind of soft/hard lock mechanism for just turning off touch and having a proper lockscreen up would be useful. The new screenlock / suspend mechanism</h2> To understand the new mechanism you first have to understand sxmo_mutex.sh. (It is documented in it's first commit.</a>) Diff from previous, old mechanism: The is_idle and can_suspend hooks were replaced by mutex locks called from various sources.</li> The crust part of sxmo_screenlock.sh was replaced by sxmo_suspend.sh and sxmo_hook_suspend.sh.</li> The hacky detect locked state approach was moved to a much cleaner have a file which holds the state approach.</li> The rest of sxmo_screenlock.sh was replaces by hook scripts which makes my journey much easier.</li> </ul> Suspending</h3> This suspend part consists of a few files: default_hooks/sxmo_hook_suspend.sh </dt> Does the actual suspending. </dd> default_hooks/sxmo_hook_mnc.sh </dt> Figures out how long it is until the next Chronjob with sxmo_rtcwake in its name is scheduled. </dd> default_hooks/sxmo_hook_presuspend.sh </dt> Called before suspending, does cleanup like pausing media and turning off screen. </dd> default_hooks/sxmo_hook_postwake.sh </dt> Called after waking up with the reason for the wakeup as the first argument. </dd> core/sxmo_suspend.sh </dt> When called fires off the whole suspend hook firework, goes too sleep, wakes up again and does a bit of housekeeping. </dd> ore/sxmo_rtcwake.sh </dt> Cronjob wrapper (I assume) that does some housekeeping with locks to make sure the cronjobs don't get interrupted. </dd> </dl> The whole thing is triggered from the screenoff hook which starts a sxmo_periodically which runs sxmo_hook_check_state_mutexes.sh and if there is no good reason to stay awake goes to sleep using sxmo_suspend.sh with a sxmo_mutex.sh and a holdexec. (At least for touchscreens with three buttons which is the category my pinephone falls into) Locking</h3> Sxmo still has the three states, but they were moved to their own hooks. This means that … The unlock script is now responsible for calling sxmo_hook_lock.sh</li> The lock script is responsible for turning the screen off</li> The screenoff script is responsible for going to deep sleep</li> Unlocking still happens through the lockstate cycling in the inputhandler</li> The unlock script is called by the start hook</li> </ul> The proximitylock script calls the screenoff hook to lock and the unlock hook, so a screenlocker can actually be placed in the lock hook without having to worry about the proximitylock interfering, the postwake hook would have to unlock the touchscreen from the screenoff hook, which it already does and the lock hook shouldn't turn off the touchscreen (that's a beautiful hook fallback), the only "problem" with this approach would be that the state would say unlocked while the lockscreen is up. (interactive would be a better identifier) To be continued</h2> Note 2022-05-07: there was a previous version of this article that said: this little diff will do it, just use the lock hook … Nope, that won't work … Most of the above is still true for the released version (good enough as a little guide for myself at least). For the actual implementation see part 2 </a> Swaylock Mobile 2022-03-03T00:00:00+00:00 Overview</h2> So far I have implemented: A flat typing indicator that is (in my opinion) a better fit for small screens</li> A numeric keypad at the bottom, currently only reacting to touch input that is not very configurable</li> </ul> Also this is my first wayland project and my first attempt at writing more than a "Hello World" in C. Log</h2> 2022-03-06 Initial release</h3> The first goal was to get a working prototype that looks acceptable (Some alignments are still off). I have reached this by implementing a flat indicator that works a bit like the ring but is a line with the width configured using the radius and if necessary with a text-box below. I was able to lock my phone by manually calling it and unlock it using the new keypad, next step is to integrate it with Sxmo so that I finally don't have to rely on the incompetence of a potential attacker for the security of my phone. 2022-03-08 schlock</h3> Apparently I'm not the first to attempt to write a lockscreen with numeric keypad based on swaylock, schlock</a>, written by @telent_net</a> appeared about two weeks ago. (Thank you for the link goes to Willow Barraco</a>) It also is targeted at the touchscreen + numeric pin usecase but our approaches under the hood seem to be completely different. While the overall approach with schlock is to replace the UI of swaylock with a virtual numpad and a custom indicator (see schlock's render.c</a> and pinentry.c</a>) the my swaylock-mobile trys to adapt and expand the UI to include touchscreen functionality so that my fork remains useful on non-touchscreens. However, the outcome is mostly the same for touchscreens. On the password matching side schlock is a bit more creative than swaylock-mobile, it seems to (I'm not sure, judging based on having looked at the commit diffs for about half an hour) have a separate code-path that matches against the hash of a pin in a file (that could be a backend and it would be awesome!) while mine works like the phosh lockscreen and simply asks pam (or whatever your swaylock backend is) if the entered pin is the users password (ab)using the existing mechanism for keyboards. There is no better here, just ideas from two screenlockers aimed at a phone that runs linux with their own ideas on how to get from a common starting point to a common goal. 2022-03-19 Integrating with sxmo 1.9.0</h3> I wrote a blogpost about how screenlocking in sxmo works</a> pre 1.9.0 and post 1.9.0 and how I'd integrate swaylock-mobile (or any other screenlocker) with it. 2023-02-26 Curently abandonned</h3> Because of some real-life happenings and some other projects I declare this as officially abandonned, I currently don't need my phone. That doesn't mean, that I won't pick it up again. 2023-05-28 sxmo_swaylock</h3> KaffeinatedKat merged the swaylock-mobile keypad into his sxmo_swaylock</a> project after I proposed it to be merged into swaylock-effects</a>. Thank you for adopting it! Yes I'm very okay with someone else adoping the code and/or idea too! My current Smarthome Setup 2021-08-24T00:00:00+00:00 Overview</h2> Features</h3> Can turn GPIOs on and off (I use that for routing audio and switching relays for power outlets)</li> Pings my machines every few seconds to find out if they are on (I have planned for when that assumption falls flat)</li> Gemini based API (for keyboard shortcuts) and dashboard</li> Scriptable using anything that can read from and write to stdio (I use lua)</li> </ul> Documentation</h3> Detailed documentation is in the README </a> Lunar Widgets - Lua scriptable Gtk Widgets 2021-07-29T00:00:00+00:00 Overview</h2> This started out as a simple program to show custom keyboards (the test.lua example is a custom keyboard for newsboat) that could be useful on a touchscreen but is now going in the general direction of Eww</a> (This project is far away from what Eww can do). The API is a bit more powerful than documented in the README, it's probably best to look at the examples and the sourcecode to find out what else can be done. I once wrote a prototype for a notification display that actually worked (notifications got fed in via stdin). DDB - Object Key Vale Store 2021-07-11T00:00:00+00:00 Overview</h2> Usage can include piping system usage information into multiple receiver scripts, storing the last wheater report for offline usage or exposing an interface for a background service. A bit like dbus but a lot simpler. Why does it exist ?</h2> There wasn't really an easy and obvious way to let two scripts in bash, lua, python or whatever talk to each other, especially in many to many communication scenarios. That's why DDB (I wanted to use it for scripts on my desktop) was written in its first version, listening on a TCP socket on localhostand granting access to a hash table and speaking a line based protocol with spaces as field seperators with (horrible) bash scripts as client implementations. If you are interested in the protocol, its documented in the README</a> and can be considered stable. Features that I needed in DDB: Introspection of some kind</li> Ability to use standard-io for most scripts (that way you don't need any fancy debugging tools)</li> A protocol wich can be implemented in a pretty simple way</li> Storage integrated into the ipc mechanism</li> Ability to subscribe to events</li> </ul> Usage</h2> I'm currently (2023-05-28</time>) using it as the message bus behind my little smarthome</a>, but it is probably going to the attic soon. Vala project setup script 2021-05-27T00:00:00+00:00 Overview</h2> When experimenting with gtk and Vala I notices that I was copy-pasting files to bootstrap new projects and experiments all the time, replacing names by hand and … making mistakes. So I put everything into a lua script that helps me with bootstrapping Vala based projects (which at the time of writing, mid 2022 I still do on occasion) Features</h2> Automatic generation of meson.build files</h3> The script generates two buildfiles for meson, one in the projects root where meson finds it and a second one in the src folder that contains a list of all the files and your dependency list you probably want to extend. It also gives you a script that you can call to update the list of files to be compiled (list of .vala</code> files in the src directory) which is good enough for most projects. One command to build and run</h3> Building with meson is a multi command process, you have to tell meson to generate the build directory then you have to tell ninja to do the actual building, find the result and run it … too many braincycles wasted for my taste, this script gives you a build</code> and a run</code> script that can do all the work for you (the run script invoking the buildscript on every call) Since you are in charge as soon as the script exits you can do whatever the hell you want with those. Generates an initial README.md wich explains how to use the build-system</h3> Probably most often asked question by everyone: How do I build that thing?</q> This thing generates a README.md skeleton for you. (you still have to fill it with information, but at least you don't have to copypasta your explanation of how to use meson) Optionally generates a hello world program</h3> Sometimes its nice to have a bit of code that reminds one of the Language syntax … , this also helps with testing the setup script itself. Sets up a git repository and makes the initial commit</h3> Another two commands that you type in everytime and are everytime the same commands, so while I was at it I automated it. Example invocation</h2> $ setup_project.lua</kbd></code> New Vala Project This program will setup a new vala project using meson ------------------------------------------------------- Please enter the name of your project > slate</kbd> Please enter the reverse domain name of the projects home > net.slatecave</kbd> Create a main.vala file? (y/N) > y</kbd> rename.sh is a script that is able to run a single sed command across every *.vala file in the src/ directory replacing anything with everything, wich is useful for renaming classes and interfaces but can also do a very excellent job at ruining your day without asking if you screw up and don't have a backup! -- If you enable it make sure you have a versioning system ready!! -- Add rename.sh? (Use a versioning/backup system!) (y/N) > y</kbd> Initialize git repository? (y/N) > y</kbd> Add a remote git repository? (y/N) > y</kbd> Ok, create your remte repo and give me the (clone) address. > https://example.org/git/slate</kbd> ------------------------------------------------------- Project Executable Name: slate Project id: net.slatecave.slate Local project folder: /home/baschdel/Development/site_generator/src/projects/vala-project-setup-script + Add a main.vala file + Add the rename.sh bulksed script + Initialize a git repository with remote https://example.org/git Is this information correct? (y/N) > N</kbd> Please restart the script then. </samp> </figure> I hope this helps people understand the code (you shouldn't let code that some random wierdo from the Internet wrote manage your projects without making sure that code does what you want). A wrapper for a terraria server with some perks 2021-05-05T00:00:00+00:00 Overview</h2> How it came into existence</h2> Long story short: A friend convinced me and a few other friends to try out Terraria</a>, we played together using his PC as the server and immedately ran into a strange problem related to us being in germany and demanding bandwidth from our ISPs. Setting up a server on a cloudserver wasn't a big problem, but I wnated something that shut the Server down when nobody used it while we were in School. So I wrote a few lines of python … and called that thing terraria-supervisor.py</code>. The other features were a: Wouldn't it be cool if?</q> First we got annoyed by the time it took to get dark (or become day again), and I wasn't very keen on logging into my Server every time so I added the cheaty part which redirected the server output to the ingame chat and in reverse read the ingame chat and looked for something command like and pipes that into the admin console using a pcall. The chatty part was me already having a Matterbridge set up for our little community (because I don't like Discord) and on the other end having a script that can read and write our ingame chat. A few tips on Game Server administration</h2> Unfortunately that server didn't last very long until it was abandonned, here is the lesson I learned: Disable cheating and bringing in stuff from the outside (and write that rule along with the reasoning down somewhere!), someone will eventually join with their level 9001 character or turn the cheatclient on, you all have a couple of hours of fun beat that one boss in no time at all and then be greeted with a gameworld that is too difficult for everyone playing fair (and boring for the others), a rollback feels like loosing progress, both options kill the world on the server and the players (including YOU, the admin) won't come back. Dragonstone - Gemini/Gopher Browser 2019-11-03T00:00:00+00:00 Overview</h2> It started as a Gopher browser inspired by the Pocket Gopher Android App, later support for the gemini and finger protocols was added. It also has theming support for documents. I use it as my primary gopher/gemini browser. A little background and history</h2> So why does someone write a browser at all?</h3> The project started as a coincidence: At the end of 2019 I lwaenewd about a protocol called gopher and started exploring some gopherholes … On an Android Phone with an app called Pocket Gopher which looked nice, was useable but had no cache at all and I wanted a desktop app anyway. At the same time I found a Vala Tutorial by Alessandro Castellani</a> and needed a project to turn all the things I've learned into something useful … While the original plan was to write a gopher browser with local file browsing capabilities I discovered the Gemini protocol</a>, and wrote some extra code to handle gemini requests</a>. Fun fact: A few days later Julien Blanchard started working on the Castor browser</a>, where I later copied the idea to use a GtkTextView for document rendering (dragonstones terrible link implementation was my idea to get rid of some performance issues with the button in textview approach). Request infrastructure</h3> Or: Why making a browser is easy but hard to get right The request infrastructure of Dragonstone was originally based on the idea that a request with an uri would be passed tp a resource store, processed and then be returned to the tab wich selects a rendering widget based on the success and file type. (simple gopher browser with the capability to view local files) Letting the widgets see (and touch, uploading in dragonstone is a giant hack) the requests was a GIGANTIC mistake, UI code should NEVER do any application logic … It also turned out (earlier actually) that thinking in therms of resource stores was also a flawed Idea, caching is required across almost all networking protocols and sometimes a resource store needs extra information like a certificate, the "session" was born a rewrite of a lot of code later to have this functionality in seperate blocks. Failed rewrites</h3> There were two attempts to rewrite Dragonstone: The Nightcat Experiment</a></li> The Slate Experiment</a></li> </ul> (Note that I'll reuse both names and I'll refer to them as experiments and eventually update the git repo. You probybly have noticed, that I reused Slate for this website and my new nick) Nightcat was an excellent Request framework, except that I realized too late that When streaming resurces there could be a result before the final result and that killed it because it was built with the assumption that the browser waits for the result with only a progress indicator. Slate on literally the other end reused Dragonstones flawed request framework in an attempt to decouple the UI from the requests entirely before eventually replacing the requests framework with a better one. It even reused Dragonstones document renderer, unfortunately this approach takes time and I put it on hold in favour of some other project. (It is unuseable in its current state, but maybe I'll pick it back up some day) A Tetris clone 2019-05-21T00:00:00+00:00 Overview</h2> It uses the cs1graphics library</a> for painting and is the project I learned python with. Information about gow to play and a little overview for Modding the game is in the README. A bit about the Engine</h2> The engine is split into two parts: The renderer which gets an array of squares the preview, score and some text for an optional overlay and draws this into a window.</li> The other part is the game mechanics part which implements the game logic.</li> </ul> The tile rotations are calculated on the fly from an array of relative square positions, so if you want to try out some crazy shapes go ahed (The function is a generator so in theory one could also generate crazy tiles on the fly) Because of the split theming (alias changing the colors) is also pretty simple. Also I remember the black-white version looking rad! A simple TicTacToe game 2018-07-07T00:00:00+00:00 Overview</h2> How to play</h2> A sceenshot of a game currently in progress, playing field rendered in ascii-art at the top, a little guide which input number corresponds with wich field and a prompt asking O for their move.</figcaption> ### ##\ /## ### ### \##\ /##/ ### ### \##\ /##/ ### ### \####/ ### ### /####\ ### ### /##/ \##\ ### ### /##/ \##\ ### ### ##/ \## ### ########################################################## ########################################################## ### ### ##\ /## ### ### \##\ /##/ ### ### \##\ /##/ ### ### \####/ ### ### /####\ ### ### /##/ \##\ ### ### /##/ \##\ ### ### ##/ \## ########################################################## ########################################################## ##\ /## ### ### \##\ /##/ ### / \ ### / \ \##\ /##/ ### / / \ \ ### / / \ \ \####/ ### | | | | ### | | | | /####\ ### | | | | ### | | | | /##/ \##\ ### | | | | ### | | | | /##/ \##\ ### \ \/ / ### \ \/ / ##/ \## ### \/ ### \/ 789 456 123 O's turn: </samp></pre> </figure> The game is pretty simple, X starts and then X and O make their moves by entering the number that corresponds to the field they want to place their marker on, if one of then gets a row of three they win, if the field fills up dbefore that happens the game is a draw. Hint: Your Numpad probably has the same layout as the field numbers!

slatecave.net

Caddy JSON log format

The 39c3 Search Engines Post

Overview</h2> To summarise it went very well and I was pretty tired after congress.</p>

The SQLite strict Guide

Modern Web Search: Not Just an Index

What this post is about</h2> This post is abut the claim of:</p>

Being a Polite Crawler on the Web

Identify Your Crawler</h2> Identify your crawler so that the Admin on the other side knows who is crawling what and why. Crude attempts at trying to look like a Browser probably won't last long. This is mainly because your crawler has a very different goal from the average website visitor.</p>

Don't Scrape what You could get Through an API</h2> If you only need specific information that is available using a well documented API, strongly consider querying that API instead of scraping web pages.</p>

Don't Crawl Things you shouldn't</h2> While crawling the things you shouldn't crawl seems interesting and appealing it really isn't. In fact you probably want to crawl even less than you are allowed to.</p> Possible reasons for a Server to advise crawlers not to crawl certain sections are:</p>

A Forgejo-Runner with Podman on Alpine

Installing the Needed Dependencies</h2> Community repository needed:</b> Some of these packages come from the alpine community repository. Uncomment the relevant line in /etc/apk/repositories</code> and run apk update</code> to enable it.</p>

Creating a User for the Runner</h2> For the runner we'll create a regular user account without any special access. For this example I'm calling the user runner-for-someone</code>, you can of course use your own name, make sure you always replace runner-for-someone</code> with that.</p>

Configuring the Forgejo Runner</h2> For this step witch into the runner user using the following command:</p> sudo</span> -</span>iu</span> runner-for-someone</span></span></code></pre>Getting a Registration Token</h3> Now is a good time to obtain a runner token …</p> … for yourself:</p>

The Runner Service</h2> Like with the Podman service this will be split into a template and an actual service.</p> Again, resulting in one service per user.</p>

The Beginning …</h2> Now you have your own runner (hopefully 😅), it's up to you what you want to with it.</p> </blockquote>

Fixing synapse 1.128.0 with SQLite

You should read: Outrage Warps Reality

Today I learned: OpenLDAP

Who is this for?</h2> This is for people who know how to configure their LDAP clients, but now want some knowledge on how to configure a server.</p>

Understanding Lua Expressions - Live Posts

Interesting Resources for building Search Engines

How Search works</h2> Articles that dive into how a Search Engine works under the hood.</p>

Thunderbird: Use S/MIME Certificates from LDAP

xdg-settings: Setting a default browser isn't that simple

Generic Browser configuration</h2> The generic browser configuration makes use of the already mentioned helper functions.</p>

Generic URL scheme configuration</h2> Generic URL scheme configuration is mostly wrapping around the already described helpers with some special treatment for the http</code> and https</code> schemes when the BROWSER</code> environment variables are set.</p>

Acronyms on the Social Internet

Why not use && as a shortcut in shell scripting

What am I Talking About</h2> If you've been shell scripting for some time you have most probably seen the following construct:</p>

Templating Zoo

Standards Zoo

Don't Scrape what You could get Through an API</h2>
If you only need specific information that is available using a well documented API, strongly consider querying that API instead of scraping web pages.</p>

The Runner Service</h2>
Like with the Podman service this will be split into a template and an actual service.</p>
Again, resulting in one service per user.</p>

Generic Browser configuration</h2>
The generic browser configuration makes use of the already mentioned helper functions.</p>

Generic URL scheme configuration</h2>
Generic URL scheme configuration is mostly wrapping around the already described helpers with some special treatment for the `http</code> and https</code> schemes when the BROWSER</code> environment variables are set.</p>`