Update from obsidian - thinkpad
Affected files: .obsidian/plugins/obsidian-omnivore/data.json Read Later/2024-02-18 - Git Tips 1- Oldies but Goodies.md Read Later/2024-02-18 - Git Tips 2- New Stuff in Git.md Read Later/2024-02-18 - Git Tips 3- Really Large Repositories.md Read Later/2024-03-26 - How Do Open Source Software Lifecycles Work-.md Read Later/2024-04-18 - Unix sockets, the basics in Rust - Emmanuel Bosquet.md Read Later/2024-05-10 - Bullet Journal in 5 Minutes a Day (for busy people).md
This commit is contained in:
parent
a6c324506a
commit
993f1f4795
7 changed files with 1166 additions and 11 deletions
22
.obsidian/plugins/obsidian-omnivore/data.json
vendored
22
.obsidian/plugins/obsidian-omnivore/data.json
vendored
|
|
@ -1,23 +1,23 @@
|
||||||
{
|
{
|
||||||
"dateHighlightedFormat": "yyyy-MM-dd HH:mm:ss",
|
"dateHighlightedFormat": "yyyy-MM-dd HH:mm:ss",
|
||||||
"dateSavedFormat": "yyyy-MM-dd HH:mm:ss",
|
"dateSavedFormat": "yyyy-MM-dd HH:mm:ss",
|
||||||
"apiKey": "",
|
"apiKey": "ec3bba50-4770-471b-99b1-9953ca523d8c",
|
||||||
"filter": "HIGHLIGHTS",
|
"filter": "ADVANCED",
|
||||||
"syncAt": "",
|
"syncAt": "2024-05-12T12:29:55",
|
||||||
"customQuery": "",
|
"customQuery": "in:archive has:highlights",
|
||||||
"template": "# {{{title}}}\n#Omnivore\n\n[Read on Omnivore]({{{omnivoreUrl}}})\n[Read Original]({{{originalUrl}}})\n\n{{#highlights.length}}\n## Highlights\n\n{{#highlights}}\n> {{{text}}} [⤴️]({{{highlightUrl}}}) {{#labels}} #{{name}} {{/labels}} ^{{{highlightID}}}\n{{#note}}\n\n{{{note}}}\n{{/note}}\n\n{{/highlights}}\n{{/highlights.length}}",
|
"template": "# {{{title}}}\n\n{{# note }}\n## Notes\n\n{{{ note }}}\n{{/ note }}\n{{#highlights.length}}\n## Highlights\n\n{{#highlights}}\n{{{text}}} \n{{#note}}\n\n> [!note]\n> {{{note}}}\n{{/note}}\n\n[source]({{{highlightUrl}}}) {{#labels}} #{{name}} {{/labels}}\n\n---\n\n{{/highlights}}\n{{/highlights.length}}\n## Original\n\n{{{ content }}}",
|
||||||
"highlightOrder": "LOCATION",
|
"highlightOrder": "LOCATION",
|
||||||
"syncing": false,
|
"syncing": false,
|
||||||
"folder": "Omnivore/{{{date}}}",
|
"folder": "Read Later/",
|
||||||
"folderDateFormat": "yyyy-MM-dd",
|
"folderDateFormat": "yyyy-MM-dd",
|
||||||
"endpoint": "https://api-prod.omnivore.app/api/graphql",
|
"endpoint": "https://api-prod.omnivore.app/api/graphql",
|
||||||
"filename": "{{{title}}}",
|
"filename": "{{{dateSaved}}} - {{{title}}}",
|
||||||
"filenameDateFormat": "yyyy-MM-dd",
|
"filenameDateFormat": "yyyy-MM-dd",
|
||||||
"attachmentFolder": "Omnivore/attachments",
|
"attachmentFolder": "Read Later/attachments",
|
||||||
"version": "1.6.3",
|
"version": "1.6.3",
|
||||||
"isSingleFile": false,
|
"isSingleFile": false,
|
||||||
"frequency": 0,
|
"frequency": 60,
|
||||||
"intervalId": 0,
|
"intervalId": 13,
|
||||||
"frontMatterVariables": [],
|
"frontMatterVariables": [],
|
||||||
"frontMatterTemplate": ""
|
"frontMatterTemplate": "id: {{{ id }}}\ntitle: >\n {{{ title }}}\nstatus: {{{ state }}}\ntags:\n - read-later\n{{#labels.length}}\n{{#labels}} - {{{name}}}\n{{/labels}}\n{{/labels.length}}\ndate_added: {{{ dateSaved}}}\nurl_omnivore: >\n {{{ omnivoreUrl}}}\nurl_original: >\n {{{ originalUrl}}}"
|
||||||
}
|
}
|
||||||
192
Read Later/2024-02-18 - Git Tips 1- Oldies but Goodies.md
Normal file
192
Read Later/2024-02-18 - Git Tips 1- Oldies but Goodies.md
Normal file
|
|
@ -0,0 +1,192 @@
|
||||||
|
---
|
||||||
|
id: a6c66f79-fd38-4d76-b05a-9c5d7dc9119f
|
||||||
|
title: |
|
||||||
|
Git Tips 1: Oldies but Goodies
|
||||||
|
status: ARCHIVED
|
||||||
|
tags:
|
||||||
|
- read-later
|
||||||
|
- dotfiles
|
||||||
|
- git
|
||||||
|
date_added: 2024-02-18 11:02:02
|
||||||
|
url_omnivore: |
|
||||||
|
https://omnivore.app/me/git-tips-1-oldies-but-goodies-18dbc861eae
|
||||||
|
url_original: |
|
||||||
|
https://blog.gitbutler.com/git-tips-1-theres-a-git-config-for-that/
|
||||||
|
---
|
||||||
|
|
||||||
|
# Git Tips 1: Oldies but Goodies
|
||||||
|
|
||||||
|
## Highlights
|
||||||
|
|
||||||
|
But there is also `--system` (which probably none of you have used) which writes it to a system-wide config file and `--local` (the default) that writes it to `.git/config` in whatever project you're currently in.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-1-oldies-but-goodies-18dbc861eae#4994949d-07aa-4baf-8b39-7b003dcc4487)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
However, there is a _secret fourth_ place that Git can look. If you add to your global config something that looks like this:
|
||||||
|
|
||||||
|
```cs
|
||||||
|
[includeIf "gitdir:~/projects/oss"]
|
||||||
|
path = ~/.gitconfig-oss
|
||||||
|
```
|
||||||
|
|
||||||
|
Then Git will look in the `~/.gitconfig-oss` files for values _only if_ the project you are currently working in matches `~/projects/oss` . So, you could have a "work" directory and have work-specific values there (company email address, gpg signing key, etc) and an "oss" directory with values for your open source projects, etc.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-1-oldies-but-goodies-18dbc861eae#400065ab-ea9d-4b03-80c8-aec98b643a27)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
One thing that is really not great about using blame in GUI tools is that the CLI has much more powerful tooling for finding something closer to the real story behind your code.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-1-oldies-but-goodies-18dbc861eae#60d94f0c-f71c-4e30-83d7-ed9d4ac07265)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Finally, if you're rebasing or cherry-picking a lot and you've ever run into the same conflict more than once, you can turn on a feature where Git memorizes the conflict and the resolution to it. If it ever sees that same conflict again, it will _automatically_ re-solve it for you.
|
||||||
|
|
||||||
|
You can easily turn it on with the config setting `rerere.enabled` and you can further ask it to automatically stage it for you with `rerere.autoUpdate`
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
$ git config --global rerere.enabled true
|
||||||
|
$ git config --global rerere.autoUpdate true
|
||||||
|
```
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-1-oldies-but-goodies-18dbc861eae#7e50290a-231d-4e52-9518-2e8ad22503cf)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Original
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Do you know some of the cool stuff in Git that's been around for a while? All the magical -L and -C options in the Git world? Let's find out!
|
||||||
|
|
||||||
|
For the first in our [short series of Git tips](https://blog.gitbutler.com/git-tips-and-tricks/), I wanted to start with stuff that's been around for a while, but it still seems like a lot of people don't know about or don't know how to use.
|
||||||
|
|
||||||
|
None of this is new, but I find them useful and they're arguably a little obscure. I'm just going to cover:
|
||||||
|
|
||||||
|
* [Conditional Configs](#conditional-configs)
|
||||||
|
* [Git Blame and Log with Line Ranges](#git-blame-and-log-with-line-ranges)
|
||||||
|
* [Git Blame with Following](https://blog.gitbutler.com/git-tips-1-theres-a-git-config-for-that/git-blame-with-following)
|
||||||
|
* [Word Diff](#word-diff)
|
||||||
|
* [Resolution Reuse](#reuse-recorded-resolution)
|
||||||
|
|
||||||
|
Let's dig in!
|
||||||
|
|
||||||
|
## Conditional Configs
|
||||||
|
|
||||||
|
Many of you probably know this, but Git has a cool little key/value store called `git config` which will check in three places for values to use when it's running various commands.
|
||||||
|
|
||||||
|
Every Git user will have [probably been told](https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup?ref=blog.gitbutler.com) to run something like this when they first set up:
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
$ git config --global user.name "John Doe"
|
||||||
|
$ git config --global user.email johndoe@example.com
|
||||||
|
```
|
||||||
|
|
||||||
|
That adds the `user.name` value to your `~.gitconfig` file. ==But there is also== `==--system==` ==(which probably none of you have used) which writes it to a system-wide config file and== `==--local==` ==(the default) that writes it to== `==.git/====config==` ==in whatever project you're currently in.==
|
||||||
|
|
||||||
|
When Git looks for a value, it will look in that order - local, global, system - for a definition.
|
||||||
|
|
||||||
|
==However, there is a== _==secret fourth==_ ==place that Git can look. If you add to your global config something that looks like this:==
|
||||||
|
|
||||||
|
```cs
|
||||||
|
[includeIf "gitdir:~/projects/oss"]
|
||||||
|
path = ~/.gitconfig-oss
|
||||||
|
```
|
||||||
|
|
||||||
|
==Then Git will look in the== `==~/.gitconfig-oss==` ==files for values== _==only if==_ ==the project you are currently working in matches== `==~====/projects/====oss==` ==. So, you could have a "work" directory and have work-specific values there (company email address, gpg signing key, etc) and an "oss" directory with values for your open source projects, etc.==
|
||||||
|
|
||||||
|
But `gitdir` is not the only filter you can use. You can also put _branch name_ specific values as a include filter with `onbranch` or you can only include config files if the project you are currently in has a remote matching a specific URL with `hasconfig:remote.*.url` . So like if you have GitHub org specific keys or something.
|
||||||
|
|
||||||
|
Check out [the docs](https://git-scm.com/docs/git-config?ref=blog.gitbutler.com#%5Fincludes) for more.
|
||||||
|
|
||||||
|
## Git Blame and Log with Line Ranges
|
||||||
|
|
||||||
|
There are a couple of interesting options that you can use with `git blame` that most people don't know about and nearly none of the existing GUIs implement.
|
||||||
|
|
||||||
|
One is the line range option, `-L`. A lot of times, if you're running blame on the command line, you just page the whole file and find the part you're looking for. However, if you want to just display a subsection of your file, you can give it a line range, like `git blame -L 28,43 path/to/file`
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
git blame -L
|
||||||
|
|
||||||
|
You can also use a semi-strange `:` syntax to give Git a pattern to find the beginning of a block and only blame that block. So in this same situation, I can get the same result by running `git blame -L :'class LocalFile' gitbutler-ui/src/lib/vbranches/types.ts`
|
||||||
|
|
||||||
|
Typically you can use a function name or something for that string.
|
||||||
|
|
||||||
|
The _other_ thing you can do to see similar information in a different way, is to run `git log` with similar options. This will give you all the commits filtered to those that last touched this region of the file. So for example, `git log -L28,43:gitbutler-ui/src/lib/vbranches/types.ts` will give you something like this:
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
So instead of being ordered by lines, it sort of gathers all the commits that are shown in the blame and then shows you those commits with code that modified that block in each commit. Basically the same data, but in a different format, more like a story of how that code was put together.
|
||||||
|
|
||||||
|
## Git Blame with Following
|
||||||
|
|
||||||
|
==One thing that is really not great about using blame in GUI tools is that the CLI has much more powerful tooling for finding something closer to the real story behind your code.==
|
||||||
|
|
||||||
|
There are many scenarios where this is really valuable. The first is ignoring whitespace changes. Some of the GUIs do that, but not all of them. So if you go and implement a `prettierrc` file, BLAM, now you're the owner of tons of lines of code. The `git blame -w` option will ignore these types of whitespace changes.
|
||||||
|
|
||||||
|
The other great option is `-C` which will look for code movement between files in a commit. So if you refactor a function from one file to another, the normal `git blame` will simply show you as the author in the new file, but the `-C` option will follow that movement and show the last person to actually change those lines of code. Either of these scenarios could be what you're looking for, but I would argue that more often it's the latter.
|
||||||
|
|
||||||
|
If you want Git to try even harder (look for movement in multiple commits or in _all_ of your commits) you can pass `-C` up to three times.
|
||||||
|
|
||||||
|
Also, your GUI _does not_ do this (most likely, I can't speak for all of them).
|
||||||
|
|
||||||
|
So, let's look at the VS Code GitLens blame output of the previous example:
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
git blame in GitLens VS Code plugin
|
||||||
|
|
||||||
|
Ok, looks good. Kiril wrote most of this code it appears. Let's now look at the same block with `git blame -w -C -C -C`
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
git blame -C -C -C
|
||||||
|
|
||||||
|
Now we can see that Git has followed this hunk of code from file to file over the course of multiple renames.
|
||||||
|
|
||||||
|
Also, Kiril only really owns a few lines of code, Mattias actually wrote big hunks of it. If we want to know about those lines, it's much better to ask Mattias rather than Kiril, as our GUI blame would suggest.
|
||||||
|
|
||||||
|
## Word Diff
|
||||||
|
|
||||||
|
This is incredibly minor, and some GUIs have nice versions of this (I find GitHub's better than what I'm about to show you, since it subtly does both) but if you _ARE_ running `git diff` on the command line and there is a line change where something small changed within it, you can change Git's default format to word-based rather than line based with the `--word-diff` option.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
normal, line-based git diff
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
super cool git diff --word-diff
|
||||||
|
|
||||||
|
## Reuse Recorded Resolution
|
||||||
|
|
||||||
|
==Finally, if you're rebasing or cherry-picking a lot and you've ever run into the same conflict more than once, you can turn on a feature where Git memorizes the conflict and the resolution to it. If it ever sees that same conflict again, it will== _==automatically==_ ==re-solve it for you.==
|
||||||
|
|
||||||
|
==You can easily turn it on with the config setting== `==rerere====.enabled==` ==and you can further ask it to automatically stage it for you with== `==rerere.====auto====Update==`
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
$ git config --global rerere.enabled true
|
||||||
|
$ git config --global rerere.autoUpdate true
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
a conflict... always remembers
|
||||||
|
|
||||||
|
Then, the next time you get a merge conflict that it's seen before, magic!
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
automatically fix it the next time
|
||||||
|
|
||||||
|
## Up Next
|
||||||
|
|
||||||
|
Again, all of this has been in Git for a while, but if you don't know... now you know.
|
||||||
|
|
||||||
|
Next up is [New Stuff in Git](https://blog.gitbutler.com/git-tips-2-new-stuff-in-git/).
|
||||||
|
|
||||||
|
### Subscribe to new posts.
|
||||||
208
Read Later/2024-02-18 - Git Tips 2- New Stuff in Git.md
Normal file
208
Read Later/2024-02-18 - Git Tips 2- New Stuff in Git.md
Normal file
|
|
@ -0,0 +1,208 @@
|
||||||
|
---
|
||||||
|
id: 3811e8c9-49ed-47bc-8302-d7dc0529d828
|
||||||
|
title: |
|
||||||
|
Git Tips 2: New Stuff in Git
|
||||||
|
status: ARCHIVED
|
||||||
|
tags:
|
||||||
|
- read-later
|
||||||
|
- dotfiles
|
||||||
|
- git
|
||||||
|
date_added: 2024-02-18 11:02:19
|
||||||
|
url_omnivore: |
|
||||||
|
https://omnivore.app/me/git-tips-2-new-stuff-in-git-18dbc865f77
|
||||||
|
url_original: |
|
||||||
|
https://blog.gitbutler.com/git-tips-2-new-stuff-in-git/
|
||||||
|
---
|
||||||
|
|
||||||
|
# Git Tips 2: New Stuff in Git
|
||||||
|
|
||||||
|
## Highlights
|
||||||
|
|
||||||
|
So, Git has created a new force pushing option called `--force-with-lease` that will essentially check that what you last pushed is still what's on the server before it will force the new branch update.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-2-new-stuff-in-git-18dbc865f77#303433c7-6c8d-45df-b8ae-729aa598255b)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
It's pretty easy to do. Just set `gpg.format` to `ssh` and tell it where your signing key is:
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
$ git config gpg.format ssh
|
||||||
|
$ git config user.signingKey ~/.ssh/id_rsa.pub
|
||||||
|
```
|
||||||
|
|
||||||
|
Now if you run `git commit -S` it will try to sign your commit with this key. If it succeeds and you upload that public key to GitHub here (under "Signing Keys"), then you'll get pretty "verified" badges on your commits:
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-2-new-stuff-in-git-18dbc865f77#6870b9f8-0e0c-4a26-8d4e-b0dd630f7260)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
provides a way to add cronjobs that run daily, hourly and weekly maintenance tasks on your Git repositories.
|
||||||
|
|
||||||
|
You can turn it on for your Git repository by simply running:
|
||||||
|
|
||||||
|
```crmsh
|
||||||
|
$ git maintenance start
|
||||||
|
```
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-2-new-stuff-in-git-18dbc865f77#0a2d2271-21f5-46e5-8f83-8fd08a82e25c)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
This means that every hour it will rebuild your commit graph and do a prefetch (we will cover these concepts in the next post), and once per day it will clean up loose objects and put them in pack-files and also repack the object directory using the `multi-pack-index` feature (read more about that in an incredible blog post from GitHub's Taylor Blau [here](https://github.blog/2021-04-29-scaling-monorepo-maintenance/?ref=blog.gitbutler.com#multi-pack-indexes)).
|
||||||
|
|
||||||
|
Basically it will just make lots of things faster in the background all the time automatically.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-2-new-stuff-in-git-18dbc865f77#cdd9aacf-451b-4437-9976-61dcbc1322aa)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Original
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
There are a bunch of new tricks that Git can do that were added in the last few years. How up to date are you?
|
||||||
|
|
||||||
|
Next up in our [3 part series](https://blog.gitbutler.com/git-tips-and-tricks/) on "Stuff you may not know about Git", we have **New Stuff**!
|
||||||
|
|
||||||
|
Here I'm going to cover 5 relatively new things in Git that you may not have heard about, because _why would you_?
|
||||||
|
|
||||||
|
We'll cover:
|
||||||
|
|
||||||
|
* [Git Branch Stuff](#some-git-branch-stuff)
|
||||||
|
* [Safe Force Pushing](#safe-force-pushing)
|
||||||
|
* [SSH Commit Signing](#ssh-commit-signing)
|
||||||
|
* [Push Signing](#push-signing)
|
||||||
|
* [Git Maintenance](#git-maintenance)
|
||||||
|
|
||||||
|
Let's dig in!
|
||||||
|
|
||||||
|
## Some Git Branch Stuff
|
||||||
|
|
||||||
|
This is pretty minor, but one thing that's always bugged me about Git is that I run `git branch` a lot to view what branches I have, but they're in the dumbest possible order (alphabetic) and there are a million of them after a while.
|
||||||
|
|
||||||
|
At some point I started naming my branches in a way to partially cope with this. Every branch would be something like `sc-0831-my-thing` meaning that the branch topic was "my thing", it was created on August 31st and the `sc` are my initials so I can group them by whose branch. It's a lot of stupid metadata to try to cram into a branch name just because of how it's listed.
|
||||||
|
|
||||||
|
However, now we can ask Git to do two things that help with this. We can ask it to sort by `objectsize`, `authordate`, `committerdate`, `creatordate`, or `taggerdate` with the `--sort` option and we can set it as a default with the `branch.sort` config setting.
|
||||||
|
|
||||||
|
So for example, if I want to sort by last commit date descending, I can run:
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
$ git config --global branch.sort -committerdate
|
||||||
|
```
|
||||||
|
|
||||||
|
And now the default will show the branch that I last committed to at the top.
|
||||||
|
|
||||||
|
💡
|
||||||
|
|
||||||
|
Important note: the `-committerdate` has a leading `-` but __not_ a double dash. It's just a negative. I've seen people mess this up and then things break.
|
||||||
|
|
||||||
|
However, now if I have a bunch of branches, that will scroll off the screen. Sad. But now Git also has a way to take a list of branches and try to split it into columns to make better use of the screen real estate. You can do this either with the new `--column` option, or with the `column.ui` setting.
|
||||||
|
|
||||||
|
Check it out:
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Nice sorted columns for my branch output
|
||||||
|
|
||||||
|
As another sort of funny thing, in order to help with this, Git implemented it's own list to column terminal command that is sort of completely independent of anything else in Git and is it's own command called `git column`.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Just in case there is anything else you need to convert into columns that isn't Git related.
|
||||||
|
|
||||||
|
## Safe Force Pushing
|
||||||
|
|
||||||
|
The next interesting thing that Git has added somewhat recently is a way to do much safer forced pushes.
|
||||||
|
|
||||||
|
Generally most of us don't love doing forced pushes, because there is always a chance that you're overwriting someone else's commits. Let's take a scenario:
|
||||||
|
|
||||||
|
* You commit and push something to GitHub
|
||||||
|
* Someone else pulls it down, commits something and pushes it back up.
|
||||||
|
* You amend a commit, rewriting the history, and force push it, not knowing that anyone had based something off your work.
|
||||||
|
* This effectively removes what the other person had done.
|
||||||
|
|
||||||
|
What you really want to do is check to see if anyone else had pushed and only force push if the answer is no. However, there is always a bit of a race condition here because even if you check first, in the second it takes you to then push, something else could have landed from elsewhere in the meantime.
|
||||||
|
|
||||||
|
==So, Git has created a new force pushing option called== `==--====force====-====with====-lease==` ==that will essentially check that what you last pushed is still what's on the server before it will force the new branch update.==
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
A failed --force-with-lease push
|
||||||
|
|
||||||
|
If someone has updated the remote ref (pushed in the meantime), then push now fails with a "stale info" error.
|
||||||
|
|
||||||
|
If you're amending and rebasing stuff a lot, it may be worth setting up a nice little alias for this, because it's almost _always_ better than running `--force`
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
$ git config --global alias.fpush push --force-with-lease
|
||||||
|
```
|
||||||
|
|
||||||
|
May the force be with you.
|
||||||
|
|
||||||
|
## Commit Signing with SSH
|
||||||
|
|
||||||
|
We [wrote about this](https://blog.gitbutler.com/signing-commits-in-git-explained/) a few months ago in mind-numbing detail, because the GitButler client does this automatically for you with the flip of a config setting, but if you want to do this on the command line, read on.
|
||||||
|
|
||||||
|
Git has supported signing your commits with GPG for a while, but GPG is often pretty difficult to get working properly and completely understand if you've never used it before. Recently, OpenSSH provided a new way to sign data using your existing SSH key and Git has integrated this as an option to use instead of GPG to do the same thing. Also, importantly, GitHub and GitLab support verifying these signatures if you upload your public signing key to your user account there.
|
||||||
|
|
||||||
|
==It's pretty easy to do. Just set== `==gpg====.format==` ==to== `==ssh==` ==and tell it where your signing key is:==
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
$ git config gpg.format ssh
|
||||||
|
$ git config user.signingKey ~/.ssh/id_rsa.pub
|
||||||
|
```
|
||||||
|
|
||||||
|
==Now if you run== `==git commit -S==` ==it will try to sign your commit with this key. If it succeeds and you upload that public key to GitHub here (under "Signing Keys"), then you'll get pretty "verified" badges on your commits:==
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Stay vigilant.
|
||||||
|
|
||||||
|
## Push Signing
|
||||||
|
|
||||||
|
I won't go into a ton of detail here because this isn't really widely used, but it might be interesting to some. Git can also now sign _pushes_, not just commits.
|
||||||
|
|
||||||
|
Since none of the major Git hosting solutions (GitHub, GitLab, Bitbucket) support this, it's only really possible to do this if you run your own server. However, if you do, you can run `git push --signed` in order to sign the ref update on the server and have the server save a transparency log with verifiable signatures somewhere.
|
||||||
|
|
||||||
|
If you're interested in this, there is a very nice [writeup](https://people.kernel.org/monsieuricon/signed-git-pushes?ref=blog.gitbutler.com) by over at kernel.org.
|
||||||
|
|
||||||
|
Push it real good.
|
||||||
|
|
||||||
|
## Git Maintenance
|
||||||
|
|
||||||
|
The final fun new thing I'll cover is `git maintenance`.
|
||||||
|
|
||||||
|
The maintenance command was introduced in Git 2.30 I believe. It essentially ==provides a way to add cronjobs that run daily, hourly and weekly maintenance tasks on your Git repositories.==
|
||||||
|
|
||||||
|
==You can turn it on for your Git repository by simply running:==
|
||||||
|
|
||||||
|
```crmsh
|
||||||
|
$ git maintenance start
|
||||||
|
```
|
||||||
|
|
||||||
|
This will modify your `.git/config` file to add a `maintenance.strategy` value set to `incremental` which is a shorthand for the following values:
|
||||||
|
|
||||||
|
* `gc`: disabled.
|
||||||
|
* `commit-graph`: hourly.
|
||||||
|
* `prefetch`: hourly.
|
||||||
|
* `loose-objects`: daily.
|
||||||
|
* `incremental-repack`: daily.
|
||||||
|
|
||||||
|
==This means that every hour it will rebuild your commit graph and do a prefetch (we will cover these concepts in the next post), and once per day it will clean up loose objects and put them in pack-files and also repack the object directory using the== `==multi-====pack====-====index==` ==feature (read more about that in an incredible blog post from GitHub's Taylor Blau== ==[here](https://github.blog/2021-04-29-scaling-monorepo-maintenance/?ref=blog.gitbutler.com#multi-pack-indexes)====).==
|
||||||
|
|
||||||
|
==Basically it will just make lots of things faster in the background all the time automatically.==
|
||||||
|
|
||||||
|
Git maintenance will schedule these cron jobs differently depending on the operating system. On Mac it will add some LaunchAgents like this:
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
If you're curious what these plist files look like, it's something like this:
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
You can read more about git maintenance and it's various options [here](https://git-scm.com/docs/git-maintenance?ref=blog.gitbutler.com).
|
||||||
|
|
||||||
|
OK, now onto our next post where we cover those commit graph and prefetching topics. Let's get into [Really Big Repositories](https://blog.gitbutler.com/git-tips-3-really-large-repositories/).
|
||||||
|
|
||||||
|
### Subscribe to new posts.
|
||||||
300
Read Later/2024-02-18 - Git Tips 3- Really Large Repositories.md
Normal file
300
Read Later/2024-02-18 - Git Tips 3- Really Large Repositories.md
Normal file
|
|
@ -0,0 +1,300 @@
|
||||||
|
---
|
||||||
|
id: 0f04cdaa-c871-4ba1-a882-7aecf65a2505
|
||||||
|
title: |
|
||||||
|
Git Tips 3: Really Large Repositories
|
||||||
|
status: ARCHIVED
|
||||||
|
tags:
|
||||||
|
- read-later
|
||||||
|
- dotfiles
|
||||||
|
- git
|
||||||
|
date_added: 2024-02-18 11:02:27
|
||||||
|
url_omnivore: |
|
||||||
|
https://omnivore.app/me/git-tips-3-really-large-repositories-18dbc867c96
|
||||||
|
url_original: |
|
||||||
|
https://blog.gitbutler.com/git-tips-3-really-large-repositories/
|
||||||
|
---
|
||||||
|
|
||||||
|
# Git Tips 3: Really Large Repositories
|
||||||
|
|
||||||
|
## Highlights
|
||||||
|
|
||||||
|
Git has had shallow clones. You could almost always run something like `git clone --depth=1` to get only the last commit and the objects it needs and then `git fetch --unshallow` to get the rest of the history later if needed. But it did break lots of things. You can't `blame`, you can't `log`, etc.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-3-really-large-repositories-18dbc867c96#d319f349-d2ea-4aed-9558-d67e4b708d74)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
If you want to do a blobless clone, you just pass `--filter=blob:none`
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-3-really-large-repositories-18dbc867c96#d296df77-d1f6-4a1b-99c6-9716824f2ee2)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
If each of those subdirectories was huge, this could be annoying to manage. Instead, we can use sparse checkouts to filter the checkouts to just specified directories.
|
||||||
|
|
||||||
|
To do this, we run `git sparse-checkout set [dir1] [dir2] ...`
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-3-really-large-repositories-18dbc867c96#d92785cc-9dde-46f8-9764-af2195722289)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
[Scalar](https://git-scm.com/docs/scalar?ref=blog.gitbutler.com) is mostly just used to clone with the correct defaults and config settings (blobless clone, no checkout by default, setting up maintenance properly, etc).
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/git-tips-3-really-large-repositories-18dbc867c96#96d0ab7b-1bf9-4b10-a08a-3d13c01bf56c)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Original
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Did you know that Git can handle enormous monorepos like Windows? Let's take a look at how.
|
||||||
|
|
||||||
|
In our third and final section of our [Git Tips series](https://blog.gitbutler.com/git-tips-and-tricks/), we're going to talk about how well Git now handles **very large** repositories and monorepos.
|
||||||
|
|
||||||
|
Do you want to use vanilla Git today to manage a 300GB repo of 3.5M files receiving a push every 20 seconds from 4000 developers without problems? **Read on!**
|
||||||
|
|
||||||
|
Here is our blog agenda. Our blogenda.
|
||||||
|
|
||||||
|
* [Prefetching](#prefetching)
|
||||||
|
* [Commit Graph](#commit-graph)
|
||||||
|
* [Filesystem Monitor](#filesystem-monitor)
|
||||||
|
* [Partial Cloning](#partial-cloning)
|
||||||
|
* [Sparse Checkouts](#sparse-checkouts)
|
||||||
|
* [Scalar](#scalar)
|
||||||
|
|
||||||
|
## First, Let's Thank Windows
|
||||||
|
|
||||||
|
Before we get started though, the first thing we have to do is thank Microsoft for nearly all of this.
|
||||||
|
|
||||||
|
In 2017, Microsoft successfully moved the Windows codebase to Git. Brian Harry wrote a really great blog post about it called [The Largest Git Repo on the Planet](https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/?ref=blog.gitbutler.com) that you should read if you're interested, but the size and scope of this repository is astounding.
|
||||||
|
|
||||||
|
* **3.5M** files
|
||||||
|
* for reference, the Linux kernel is about 80k files, or 2% of that
|
||||||
|
* **300GB** repository (vs \~4.5G Linux kernel)
|
||||||
|
* **4,000** active developers
|
||||||
|
* **8,421** pushes per day (on average)
|
||||||
|
* **4,352** active topic branches
|
||||||
|
|
||||||
|
In order to get that to work in any possible way, Microsoft had a lot of work to do. With vanilla Git at that time, a lot of commands (ie, `git status`) would take hours if they ever finished at all. They needed to make every command close to as fast as Source Depot was.
|
||||||
|
|
||||||
|
The first solution to this problem was a new project called [VFS for Git](https://github.com/microsoft/VFSForGit?ref=blog.gitbutler.com) which was a virtual filesystem layer that did virtual checkouts and then requested files from a central server on demand.
|
||||||
|
|
||||||
|
Eventually they moved more and more of the solutions they developed to the [Scalar](https://github.com/microsoft/scalar?ref=blog.gitbutler.com) project, which got rid of the virtualization layer, and would instead request file contents on checkout rather than on demand.
|
||||||
|
|
||||||
|
Then they moved everything, piece by piece, into the [Microsoft Git](https://github.com/microsoft/git?ref=blog.gitbutler.com) fork and then finally every part of _that_ was moved into [core Git](https://git-scm.com/docs/scalar?ref=blog.gitbutler.com).
|
||||||
|
|
||||||
|
So, as promised, if you want to use Git out of the box today to manage a 300GB repo of 3.5M files receiving a push every 20 seconds from 4000 developers, you can _100%_ do so.
|
||||||
|
|
||||||
|
Let's get into everything they added for us.
|
||||||
|
|
||||||
|
## Prefetching
|
||||||
|
|
||||||
|
So, in the last blog post I talked about how running `git maintenance` on a repository will run prefetching and maintain your commit graph every hour. Let's cover the first of those. What is "prefetching"?
|
||||||
|
|
||||||
|
One of the things that the Windows devs found annoying was that fetches would often be slow because there was _so much_ activity going on all the time. Whenever they would fetch, they have to get _all_ the data since whatever the last time they manually fetched.
|
||||||
|
|
||||||
|
So in the cronjob, they added something called "prefetching", which will essentially run a fetch command every hour automatically for you.
|
||||||
|
|
||||||
|
However, it does not update your remote references like it normally would, instead it populates a special `refs/prefetch` area of your references.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
These references aren't shown normally, even if you run something like `git log --all`, they are quite hidden from you. However, they are used in the remote server negotiation, which means that if you have this turned on, whenever you go to fetch, your computer is never more than 1 hour of pushes behind, data-wise.
|
||||||
|
|
||||||
|
Basically it makes manual fetches fast.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Joke stolen from Martin Woodward :)
|
||||||
|
|
||||||
|
## Commit Graph
|
||||||
|
|
||||||
|
The other thing that `git maintenance` does every hour is update your commit graph data. What does that mean exactly?
|
||||||
|
|
||||||
|
Well, essentially, it makes walking your commit history faster.
|
||||||
|
|
||||||
|
Instead of opening up one commit object at a time to see what it's parent is, then opening up that object to see what it's parent is, etc, the commit graph is basically an index of all that information that can be quickly read in one go. This makes things like `git log --graph` or `git branch --contains` much, much faster. For most repositories this probably isn't a horrible problem, but when you start getting into millions of commits, it makes a huge difference.
|
||||||
|
|
||||||
|
Here's a benchmark of some log related subcommands run on the Linux kernel codebase with and without the commit graph data (from the [GitHub blog](https://github.blog/2022-08-30-gits-database-internals-ii-commit-history-queries/?ref=blog.gitbutler.com#the-commit-graph))
|
||||||
|
|
||||||
|
| **Command** | **Withoutcommit-graph** | **Withcommit-graph** |
|
||||||
|
| ------------------------- | ----------------------- | -------------------- |
|
||||||
|
| git rev-list v5.19 | 6.94s | 0.98s |
|
||||||
|
| git rev-list v5.0..v5.19 | 2.51s | 0.30s |
|
||||||
|
| git merge-base v5.0 v5.19 | 2.59s | 0.24s |
|
||||||
|
|
||||||
|
Here is a quick test that I did on the same Linux repo running `git log --graph --oneline` before and after writing a commit graph file.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Again, if you have your `maintenance` cron jobs running, this is just magically done for you constantly, you don't really have to do anything explicit.
|
||||||
|
|
||||||
|
## Filesystem Monitor
|
||||||
|
|
||||||
|
One of the things that VFS for Git needed was a filesystem monitor so that it could detect when virtual file contents were being requested and fetch them from a central server if needed.
|
||||||
|
|
||||||
|
This monitor was eventually utilized to speed up the `git status` command by updating the index based on filesystem modification events rather than running `stat` on every file, every time when you run it.
|
||||||
|
|
||||||
|
While the former became unnecessary when the virtualization layer was abandoned, the latter was integrated into Git core. If you want much, much faster `git status` runs for very large working directories, the new Git filesystem monitor is a lifesaver.
|
||||||
|
|
||||||
|
Basically you just set these config settings:
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
$ git config core.fsmonitor true
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
What this will do is add a setting that the `git status` command will see when it runs, indicating that it should be using the [fsmonitor-daemon](https://git-scm.com/docs/git-fsmonitor--daemon?ref=blog.gitbutler.com). If this daemon is not running, it will start it.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
fsmonitor on Chromium makes status 20x faster
|
||||||
|
|
||||||
|
So, the first time you run `status` after setting the config setting, it won't be much faster. But every time after that it will be massively faster.
|
||||||
|
|
||||||
|
Again, there's nothing really to explicitly do after setting the value, things just get faster.
|
||||||
|
|
||||||
|
## Partial Cloning
|
||||||
|
|
||||||
|
Another big issue with large repositories is clone size. As you probably know, by default Git will fetch everything. You don't even need to have a 300GB repository for this to be a problem. [Linux](https://github.com/torvalds/linux?ref=blog.gitbutler.com) is over 4GB, [Chromium](https://github.com/chromium/chromium?ref=blog.gitbutler.com) is over 20GB. A full Chromium clone can easily take an hour, even over a pretty fast connection.
|
||||||
|
|
||||||
|
Now, for a long time ==Git has had shallow clones. You could almost always run something like== `==git== ==clone== ==--depth=========1==` ==to get only the last commit and the objects it needs and then== `==git== ==fetch== ==--unshallow==` ==to get the rest of the history later if needed. But it did break lots of things. You can't== `==blame==`==, you can't== `==log==`==, etc.==
|
||||||
|
|
||||||
|
However, now Git has both blobless and treeless clones. So you do get the whole history (all of the commits), but you don't locally have the actual content. Let's ignore the treeless clones for now because it's not generally recommended, but the blobless clone is.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
A full clone of the Linux repository is 4.6G, or (for me) a 20 min process
|
||||||
|
|
||||||
|
==If you want to do a blobless clone, you just pass== `==--filter=====blob:none==`
|
||||||
|
|
||||||
|
This will change the process a little. It will download all the commits and trees, which in the case of cloning the Linux kernel reduces 4.6G to 1.4G, and it will then do a _second_ object fetch for just the blobs that it needs in order to populate the checkout.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
So you can see that instead of 20 minutes for the clone, it took me 4.5 minutes. You can also see that it did two fetches, one for the 1.4GBs of commit and tree data, and a second for the 243MB of files it needs for my local checkout.
|
||||||
|
|
||||||
|
Now, there are downsides to this too. If you want to run a command that needs data that is not there, Git will have to go back to the server and request those objects. Luckily, it does this on-demand as it needs the objects, but it can make something like `blame` do a bunch of roundtrips.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
round and round we go
|
||||||
|
|
||||||
|
In the case of Linux, my tests showed that a normal file blame might take 4 seconds now takes 45 seconds. But that's only the first time you need to do it.
|
||||||
|
|
||||||
|
## Sparse Checkouts
|
||||||
|
|
||||||
|
The last big thing to look at is not only useful for large repositories, but specifically for monorepos. That is, repositories that contain multiple projects in subdirectories.
|
||||||
|
|
||||||
|
For example, at GitButler, our web services are in a monorepo, with each service we run on AWS in a subdirectory.
|
||||||
|
|
||||||
|
```reasonml
|
||||||
|
❯ tree -L 1
|
||||||
|
.
|
||||||
|
├── Gemfile
|
||||||
|
├── Gemfile.lock
|
||||||
|
├── README.md
|
||||||
|
├── auth-proxy
|
||||||
|
├── butler
|
||||||
|
├── chain
|
||||||
|
├── check.rb
|
||||||
|
├── copilot
|
||||||
|
└── git
|
||||||
|
|
||||||
|
6 directories, 4 files
|
||||||
|
```
|
||||||
|
|
||||||
|
==If each of those subdirectories was huge, this could be annoying to manage. Instead, we can use sparse checkouts to filter the checkouts to just specified directories.==
|
||||||
|
|
||||||
|
==To do this, we run== `==git== ==sparse-checkout== ==set [dir1] [dir2] ...==`
|
||||||
|
|
||||||
|
```smali
|
||||||
|
❯ git sparse-checkout set butler copilot
|
||||||
|
❯ tree -L 1
|
||||||
|
.
|
||||||
|
├── Gemfile
|
||||||
|
├── Gemfile.lock
|
||||||
|
├── README.md
|
||||||
|
├── butler
|
||||||
|
├── check.rb
|
||||||
|
└── copilot
|
||||||
|
```
|
||||||
|
|
||||||
|
So we still have the top level files, but only the two subdirectories that we specified. This is called "cone mode" and tends to be pretty fast. It also makes `status` and related commands faster because there are fewer files to care about. You can also however, set patterns rather than subdirectories, but it's more complicated.
|
||||||
|
|
||||||
|
Here's a local test I did with the Chromium repository:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
❯ time git status
|
||||||
|
On branch main
|
||||||
|
Your branch is up to date with 'origin/main'.
|
||||||
|
|
||||||
|
nothing to commit, working tree clean
|
||||||
|
|
||||||
|
real 0m5.931s
|
||||||
|
|
||||||
|
❯ git sparse-checkout set build base
|
||||||
|
|
||||||
|
❯ time git status
|
||||||
|
On branch main
|
||||||
|
Your branch is up to date with 'origin/main'.
|
||||||
|
|
||||||
|
You are in a sparse checkout with 2% of tracked files present.
|
||||||
|
|
||||||
|
nothing to commit, working tree clean
|
||||||
|
|
||||||
|
real 0m0.386s
|
||||||
|
```
|
||||||
|
|
||||||
|
This is without the `fsmonitor` stuff. You can see that `status` went from 6 seconds to run down to 0.3 seconds because there just aren't as many files.
|
||||||
|
|
||||||
|
If you're using large monorepos, this means you can do a blobless clone to have a much smaller local database (you can also run `clone --no-checkout` to skip the initial checkout), then do a `sparse-checkout` to filter to the directories you need and everything is massively faster.
|
||||||
|
|
||||||
|
## Scalar
|
||||||
|
|
||||||
|
Finally, Git now (since Oct 2022, Git 2.38) ships with an _alternative_ command line invocation that wraps some of this stuff.
|
||||||
|
|
||||||
|
This command is called `scalar`. Just go ahead and type it:
|
||||||
|
|
||||||
|
```fsharp
|
||||||
|
❯ scalar
|
||||||
|
usage: scalar [-C <directory>] [-c <key>=<value>] <command> [<options>]
|
||||||
|
|
||||||
|
Commands:
|
||||||
|
clone
|
||||||
|
list
|
||||||
|
register
|
||||||
|
unregister
|
||||||
|
run
|
||||||
|
reconfigure
|
||||||
|
delete
|
||||||
|
help
|
||||||
|
version
|
||||||
|
diagnose
|
||||||
|
```
|
||||||
|
|
||||||
|
==[Scalar](https://git-scm.com/docs/scalar?ref=blog.gitbutler.com)== ==is mostly just used to clone with the correct defaults and config settings (blobless clone, no checkout by default, setting up maintenance properly, etc).==
|
||||||
|
|
||||||
|
If you are managing large repositories, cloning with this negates the need to run `git maintenance start` and send the `--no-checkout` command and remember `--filter=tree:0` and whatnot.
|
||||||
|
|
||||||
|
Now you're ready to scale! ...-ar.
|
||||||
|
|
||||||
|
## Some More Reading
|
||||||
|
|
||||||
|
If you want to read about all of this in great detail, GitHub has done an amazing job covering lots of this too:
|
||||||
|
|
||||||
|
* [The Commit Graph](https://github.blog/2022-08-30-gits-database-internals-ii-commit-history-queries/?ref=blog.gitbutler.com#the-commit-graph)
|
||||||
|
* [Sparse checkout](https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/?ref=blog.gitbutler.com)
|
||||||
|
* [Filesystem Monitor](https://github.blog/2022-06-29-improve-git-monorepo-performance-with-a-file-system-monitor/?ref=blog.gitbutler.com)
|
||||||
|
* [Sparse index](https://github.blog/2021-11-10-make-your-monorepo-feel-small-with-gits-sparse-index/?ref=blog.gitbutler.com)
|
||||||
|
* [Partial and shallow clone](https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/?ref=blog.gitbutler.com)
|
||||||
|
* [The Story of Scalar](https://github.blog/2022-10-13-the-story-of-scalar/?ref=blog.gitbutler.com)
|
||||||
|
|
||||||
|
There is also a ton more fascinating information on this from [Derrick Stolee](https://stolee.dev/?ref=blog.gitbutler.com), who did a lot of work on these projects.
|
||||||
|
|
||||||
|
Ok, that's it for our Git Tips series! Hope you enjoyed it and let us know in Discord if you have any questions or comments, or would like to see us do any other topics in Git land.
|
||||||
|
|
||||||
|
Thanks!
|
||||||
|
|
||||||
|
### Subscribe to new posts.
|
||||||
|
|
@ -0,0 +1,166 @@
|
||||||
|
---
|
||||||
|
id: 7ec51414-ec16-11ee-9d24-0f20db024fdd
|
||||||
|
title: |
|
||||||
|
How Do Open Source Software Lifecycles Work?
|
||||||
|
status: ARCHIVED
|
||||||
|
tags:
|
||||||
|
- read-later
|
||||||
|
- RSS
|
||||||
|
date_added: 2024-03-26 14:01:50
|
||||||
|
url_omnivore: |
|
||||||
|
https://omnivore.app/me/how-do-open-source-software-lifecycles-work-18e7f16d0ba
|
||||||
|
url_original: |
|
||||||
|
https://www.freecodecamp.org/news/understanding-open-source-software-lifecycles/
|
||||||
|
---
|
||||||
|
|
||||||
|
# How Do Open Source Software Lifecycles Work?
|
||||||
|
|
||||||
|
## Highlights
|
||||||
|
|
||||||
|
* There's the **alpha release** – an initial version of software that is typically not feature-complete and is not intended for use by the general public. It is used for testing and internal use only.
|
||||||
|
* A **beta release** is a pre-release version of software that is feature-complete but may still have bugs or other issues. It is released to a limited audience for testing and feedback before the final release.
|
||||||
|
* Next will be a **release candidate**, which is a version of software that is considered stable and ready for release, pending final testing and bug fixes.
|
||||||
|
* And finally you'll produce a **general availability release** as the final version of software that's released to the general public.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/how-do-open-source-software-lifecycles-work-18e7f16d0ba#e773fb30-19c5-435e-b694-3dc361102961)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Major releases are typically used for significant changes or new features that are not backward compatible with previous versions. Major releases are usually announced to users and customers with a lot of fanfare, as they represent a significant milestone in the development of the software.
|
||||||
|
|
||||||
|
Minor releases, on the other hand, are used for smaller changes or new features that are backward compatible with previous versions. Minor releases are typically released more frequently and are intended to provide users with incremental improvements to the software.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/how-do-open-source-software-lifecycles-work-18e7f16d0ba#a3fc7d71-2005-47d8-ae20-ad8ba83d0096)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Backward compatibility is the ability of a newer version of software or system to work with files, data, and other components created in an older version of that software or system. This means that users can upgrade to the newer version without losing access to their existing data or files.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/how-do-open-source-software-lifecycles-work-18e7f16d0ba#a2e6d179-5650-4258-aeca-c97135d5cbd1)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Feature freeze is a stage in the software development process where no new features are added to the product or project. It is typically implemented as a deadline by which all new features must be completed and approved before the release of the software product.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/how-do-open-source-software-lifecycles-work-18e7f16d0ba#8f0133c9-d2fd-4bd5-ad62-6e5bbb42fd39)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
A roadmap is a high-level strategic document that outlines the goals, objectives, and timeline for a software product's development. It provides a visual representation of the product development plan, outlining key milestones and the expected timeline for completion.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/how-do-open-source-software-lifecycles-work-18e7f16d0ba#15a5ff9c-9ba1-423a-9ee3-a4e9359ee251)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
A changelog is a document that lists the changes made to a software product over time, including bug fixes, new features, and other updates. Changelogs allow developers and other stakeholders to understand what's been updated and when.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/how-do-open-source-software-lifecycles-work-18e7f16d0ba#49c22f66-4cae-4b35-9830-8ca179e57c1c)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Long term support refers to a software version that is designated for longer-term support and maintenance, typically for a period of several years. During this time, the software vendor provides ongoing support, including bug fixes, security updates, and other maintenance activities.
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/how-do-open-source-software-lifecycles-work-18e7f16d0ba#44327f2b-0570-483f-9aa4-d73ba0a88d18)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
end-of-life refers to a point in time when a software version is no longer supported by the vendor. This means that the vendor will no longer provide
|
||||||
|
|
||||||
|
[source](https://omnivore.app/me/how-do-open-source-software-lifecycles-work-18e7f16d0ba#88586f0a-919d-4ce9-b50a-dc2c4176950e)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Original
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Software projects follow identifiable milestones as they move towards a successful completion. If you want to give your project the best chances of success, it's important to understand what those milestones mean and how they're defined.
|
||||||
|
|
||||||
|
This article comes from my Complete LPI Open Source Essentials Exam Study Guide [Udemy course](https://www.udemy.com/course/complete-lpi-open-source-essentials-exam-study-guide/?referralCode=05B999CE18EF4D6E243C) and [book](https://www.amazon.com/dp/B0CK3Q8DCF). You can also [view the video version](https://youtu.be/eZ%5F4DLVxs7Q).
|
||||||
|
|
||||||
|
## What are Software Releases?
|
||||||
|
|
||||||
|
There are several types of software releases and some related versioning methods used to keep track of software changes and to communicate them to users. We'll start with releases.
|
||||||
|
|
||||||
|
* ==There's the== **==alpha release==** ==– an initial version of software that is typically not feature-complete and is not intended for use by the general public. It is used for testing and internal use only.==
|
||||||
|
* ==A== **==beta release==** ==is a pre-release version of software that is feature-complete but may still have bugs or other issues. It is released to a limited audience for testing and feedback before the final release.==
|
||||||
|
* ==Next will be a== **==release candidate==**==, which is a version of software that is considered stable and ready for release, pending final testing and bug fixes.==
|
||||||
|
* ==And finally you'll produce a== **==general availability release==** ==as the final version of software that's released to the general public.==
|
||||||
|
|
||||||
|
## What is Software Versioning?
|
||||||
|
|
||||||
|
Software versioning (sometimes known as semantic versioning) is the practice of assigning unique version numbers to different releases of software.
|
||||||
|
|
||||||
|
Here's a useful example:
|
||||||
|
|
||||||
|
```angelscript
|
||||||
|
vmlinuz-5.19.0-40-generic
|
||||||
|
```
|
||||||
|
|
||||||
|
In some approaches, the first number in the version number ("5" in this case) is the major version. A major version change indicates significant changes or new features that are not backward compatible with previous versions.
|
||||||
|
|
||||||
|
The second number ("19") is the minor version. A minor version change indicates new features or functionality that are backward compatible with previous versions.
|
||||||
|
|
||||||
|
The third number in the version number ("0") is the patch version. A patch version change indicates bug fixes or minor changes that are backward compatible with previous versions.
|
||||||
|
|
||||||
|
Why distinguish between major and minor releases? ==Major releases are typically used for significant changes or new features that are not backward compatible with previous versions. Major releases are usually announced to users and customers with a lot of fanfare, as they represent a significant milestone in the development of the software.==
|
||||||
|
|
||||||
|
==Minor releases, on the other hand, are used for smaller changes or new features that are backward compatible with previous versions. Minor releases are typically released more frequently and are intended to provide users with incremental improvements to the software.==
|
||||||
|
|
||||||
|
## What Does Backward Compatibility Mean?
|
||||||
|
|
||||||
|
==Backward compatibility is the ability of a newer version of software or system to work with files, data, and other components created in an older version of that software or system. This means that users can upgrade to the newer version without losing access to their existing data or files.==
|
||||||
|
|
||||||
|
For example, let's assume a user has created a document in an older version of a word processing program. If the newer version of the program is backward compatible, the user can open and edit the same document without any issues. This is because the newer version of the program is designed to read and interpret the file format used in the older version.
|
||||||
|
|
||||||
|
However, if the newer version of the program is not backward compatible, the user may not be able to open or edit the file created in the older version without first converting or re-creating it in the newer version. This can be a significant inconvenience for users and can lead to compatibility issues and data loss.
|
||||||
|
|
||||||
|
Here are some more quick – but important – definitions.
|
||||||
|
|
||||||
|
## Feature Freeze
|
||||||
|
|
||||||
|
==Feature freeze is a stage in the software development process where no new features are added to the product or project. It is typically implemented as a deadline by which all new features must be completed and approved before the release of the software product.==
|
||||||
|
|
||||||
|
The primary goal of a feature freeze is to stabilize the software product in preparation for release. By setting a feature freeze deadline, developers can focus on completing and testing existing features rather than introducing new ones. This allows time for rigorous testing and bug fixing, improving the overall quality and reliability of the software product.
|
||||||
|
|
||||||
|
## Roadmaps
|
||||||
|
|
||||||
|
==A roadmap is a high-level strategic document that outlines the goals, objectives, and timeline for a software product's development. It provides a visual representation of the product development plan, outlining key milestones and the expected timeline for completion.==
|
||||||
|
|
||||||
|
Roadmaps are useful for communicating the overall direction of a software product to stakeholders, including developers, product managers, investors, and customers.
|
||||||
|
|
||||||
|
## Milestones
|
||||||
|
|
||||||
|
Milestones are specific, measurable achievements that mark progress towards the completion of a software product. They're typically set at regular intervals throughout the development process and are used to track progress and ensure that the project stays on schedule.
|
||||||
|
|
||||||
|
Examples of milestones might include the completion of a specific feature, the successful completion of a testing phase, or the release of a beta version of the software product.
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
==A changelog is a document that lists the changes made to a software product over time, including bug fixes, new features, and other updates. Changelogs allow developers and other stakeholders to understand what's been updated and when.==
|
||||||
|
|
||||||
|
Changelogs are particularly useful for software products that are updated frequently or have a large number of contributors.
|
||||||
|
|
||||||
|
## Long Term Support (LTS)
|
||||||
|
|
||||||
|
==Long term support refers to a software version that is designated for longer-term support and maintenance, typically for a period of several years. During this time, the software vendor provides ongoing support, including bug fixes, security updates, and other maintenance activities.==
|
||||||
|
|
||||||
|
LTS versions are often used in enterprise environments where stability and reliability are critical. In April of each even year, for example, Canonical will release an LTS version of Ubuntu. These versions are normally supported for four or five years.
|
||||||
|
|
||||||
|
## End of Life (EOL)
|
||||||
|
|
||||||
|
On the other hand, ==end-of-life refers to a point in time when a software version is no longer supported by the vendor. This means that the vendor will no longer provide== updates or fixes for the software, and any security vulnerabilities or bugs that are discovered will not be addressed. This can leave users with unsupported software that may be prone to security risks and other issues.
|
||||||
|
|
||||||
|
When a software product reaches its end-of-life, it is typically retired, and users are encouraged to upgrade to a newer version or switch to a different product. The EOL process is often gradual, with the vendor providing advance notice and guidance to users to help them migrate to a new version or product.
|
||||||
|
|
||||||
|
## Conclusion
|
||||||
|
|
||||||
|
You've seen how it's important to understand the stages through which successful software projects will move. And this isn't just theoretical, because this knowledge gives you the tools to track your progress and quickly identify when things are going off rails.
|
||||||
|
|
||||||
|
__This article comes from my [_Complete LPI Open Source Essentials Study Guide course](https://www.udemy.com/course/complete-lpi-open-source-essentials-exam-study-guide/?referralCode=05B999CE18EF4D6E243C)_. And there's much more technology goodness available at [bootstrap-it.com](https://bootstrap-it.com/)_
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Learn to code for free. freeCodeCamp's open source curriculum has helped more than 40,000 people get jobs as developers. [Get started](https://www.freecodecamp.org/learn/)
|
||||||
|
|
@ -0,0 +1,239 @@
|
||||||
|
---
|
||||||
|
id: 3f72e613-eb8e-446b-93b5-fdb275a781c0
|
||||||
|
title: |
|
||||||
|
Unix sockets, the basics in Rust - Emmanuel Bosquet
|
||||||
|
status: ARCHIVED
|
||||||
|
tags:
|
||||||
|
- read-later
|
||||||
|
date_added: 2024-04-18 20:15:21
|
||||||
|
url_omnivore: |
|
||||||
|
https://omnivore.app/me/unix-sockets-the-basics-in-rust-emmanuel-bosquet-18ef3b52a37
|
||||||
|
url_original: |
|
||||||
|
https://emmanuelbosquet.com/2022/whatsaunixsocket/
|
||||||
|
---
|
||||||
|
|
||||||
|
# Unix sockets, the basics in Rust - Emmanuel Bosquet
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
This will be useful someday...
|
||||||
|
when I want to try to build my own TCP server or something like that
|
||||||
|
## Original
|
||||||
|
|
||||||
|
## Contents
|
||||||
|
|
||||||
|
* [What is a unix socket?](#what-is-a-unix-socket)
|
||||||
|
* [Create a socket, server side](#create-a-socket-server-side)
|
||||||
|
* [Waiting for connections, server side](#waiting-for-connections-server-side)
|
||||||
|
* [Connecting to the socket, client side](#connecting-to-the-socket-client-side)
|
||||||
|
* [Writing on the socket, client side](#writing-on-the-socket-client-side)
|
||||||
|
* [Reading from the socket, server side](#reading-from-the-socket-server-side)
|
||||||
|
* [Launch the whole thing!](#launch-the-whole-thing)
|
||||||
|
* [Respond to a message, server side](#respond-to-a-message-server-side)
|
||||||
|
* [Listen to responses, client side](#listen-to-responses-client-side)
|
||||||
|
* [Launch the whole thing, again!](#launch-the-whole-thing-again)
|
||||||
|
* [Browse the code](#browse-the-code)
|
||||||
|
|
||||||
|
I found myself wondering about unix sockets while working on [Sōzu](https://github.com/sozu-proxy/sozu), a reverse proxy written in Rust. A bunch of Sōzu issues led me to[dig into Sōzu channels](https://github.com/Keksoj/stream%5Fstuff%5Fon%5Fa%5Fsozu%5Fchannel), which themselves make use of[Metal I/O ’s implementation of unix sockets](https://tokio-rs.github.io/mio/doc/mio/net/struct.UnixListener.html).
|
||||||
|
|
||||||
|
Here are the questions, summed up:
|
||||||
|
|
||||||
|
* what are unix sockets?
|
||||||
|
* how can we create them in Rust?
|
||||||
|
* how do we use them to stream data?
|
||||||
|
|
||||||
|
So here we go.
|
||||||
|
|
||||||
|
It is _not_ a web socket like `127.0.0.1:8080`.
|
||||||
|
|
||||||
|
You may have heard that in unix,[everything is a file](https://www.youtube.com/watch?v=dDwXnB6XeiA). Unix sockets seem to be a good example of this principle. They are empty files of sorts, only there to be written to, and read from.
|
||||||
|
|
||||||
|
Sockets are a core feature of unix. In fact, if you type
|
||||||
|
|
||||||
|
```ebnf
|
||||||
|
man unix
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
in your terminal, you should land on an ancient man page:
|
||||||
|
|
||||||
|
| 1 2 3 4 5 | UNIX(7) Linux Programmer's Manual UNIX(7) NAME unix - sockets for local interprocess communication |
|
||||||
|
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
|
||||||
|
that explains how sockets are declared in C in the kernel, how they are created with the `AF_UNIX` system call, and many more thing that go far beyond my limited understanding.
|
||||||
|
|
||||||
|
Creating a socket is not as easy as creating just any file, using, say, `touch`. They are tools available in the command line, but most of the time, sockets are created and used by processes, not by users. Looking up how to create one will land you on a tutorial in C, or in python. So let’s see how to do it in Rust.
|
||||||
|
|
||||||
|
The Rust standard library has a [std::os::unix module](https://doc.rust-lang.org/std/os/unix/index.html)to interact with unix processes, unix files, and so on. Within it, we want to look at the `net` module, named that way because unix sockets are used to do networking between processes.
|
||||||
|
|
||||||
|
The `std::os::unix::net` module contains, among other things:
|
||||||
|
|
||||||
|
* [UnixListener](https://doc.rust-lang.org/std/os/unix/net/struct.UnixListener.html)
|
||||||
|
* [UnixStream](https://doc.rust-lang.org/std/os/unix/net/struct.UnixStream.html)
|
||||||
|
|
||||||
|
Both those entities are unsafe wrappers of the `libc` library to perform the very same unix system calls you would write in C. They both wrap a unix file descriptor, but they are distinct in order to separate higher-level concerns.
|
||||||
|
|
||||||
|
* `UnixListener` is used to create sockets, (`libc::bind()` and `libc::listen()`)
|
||||||
|
* `UnixStream` is there to connect to a socket (`libc::connect()`), to read from it and write on it.
|
||||||
|
|
||||||
|
Let’s use those.[Install Rust and Cargo](https://www.rust-lang.org/tools/install),[Learn the basics of Rust](https://doc.rust-lang.org/book/), and then do:
|
||||||
|
|
||||||
|
```haxe
|
||||||
|
cargo new unix_sockets
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
Add this to `Cargo.toml` (makes error propagation easier):
|
||||||
|
|
||||||
|
| 1 2 | \# Cargo.toml anyhow \= "^1.0.42" |
|
||||||
|
| --- | --------------------------------- |
|
||||||
|
|
||||||
|
In the `src` directory, create a `bin` directory, in which you will create a `server.rs` file.
|
||||||
|
|
||||||
|
| 1 2 3 4 5 6 7 8 9 10 11 12 13 | // src/bin/server.rs use std::os::unix::net::{UnixListener, UnixStream}; use anyhow::Context; fn main() -> anyhow::Result<()> { let socket\_path = "mysocket"; let unix\_listener = UnixListener::bind(socket\_path).context("Could not create the unix socket")?; Ok(()) } |
|
||||||
|
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
|
||||||
|
Then do
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
cargo run --bin server
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
Which should run smoothly, and then do `ls -l` in your directory, you should have a line like this:
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
srwxr-xr-x 1 emmanuel users 0 Jan 7 13:08 mysocket
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
The `s` stands for _socket_. Congratulations!
|
||||||
|
|
||||||
|
Do one more `cargo run --bin server` and you have a neat, self-explanatory OS error:
|
||||||
|
|
||||||
|
| 1 2 3 4 | Error: Could not create the unix socket Caused by: Address already in use (os error 98) |
|
||||||
|
| ------- | ------------------------------------------------------------------------------------------- |
|
||||||
|
|
||||||
|
I guess we’ll have to destroy it and recreate it each time.
|
||||||
|
|
||||||
|
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | // src/bin/server.rs use std::os::unix::net::{UnixListener, UnixStream}; use anyhow::Context; fn main() -> anyhow::Result<()> { let socket\_path = "mysocket"; // copy-paste this and don't think about it anymore // it will be hidden from there on if std::fs::metadata(socket\_path).is\_ok() { println!("A socket is already present. Deleting..."); std::fs::remove\_file(socket\_path).with\_context(\|| { format!("could not delete previous socket at {:?}", socket\_path) })?; } let unix\_listener = UnixListener::bind(socket\_path).context("Could not create the unix socket")?; Ok(()) } |
|
||||||
|
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
|
||||||
|
The `UnixListener` struct has an `accept()` method that waits for other processes to connect to the socket. Once a connections comes, `accept()` returns a tuple containing a `UnixStream` and a `SocketAddr`.
|
||||||
|
|
||||||
|
As mentioned above, `UnixStream` implements `Read` and `Write`. We will handle this stream to:
|
||||||
|
|
||||||
|
* read what another process will send through the socket
|
||||||
|
* write responses on the socket
|
||||||
|
|
||||||
|
Add the loop and the `handle_stream` function to the server code:
|
||||||
|
|
||||||
|
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | // src/bin/server.rs use std::os::unix::net::{UnixListener, UnixStream}; fn main() -> anyhow::Result<()> { let socket\_path = "mysocket"; let unix\_listener = UnixListener::bind(socket\_path).context("Could not create the unix socket")?; // put the server logic in a loop to accept several connections loop { let (mut unix\_stream, socket\_address) = unix\_listener .accept() .context("Failed at accepting a connection on the unix listener")?; handle\_stream(unix\_stream)?; } Ok(()) } fn handle\_stream(mut stream: UnixStream) -> anyhow::Result<()> { // to be filled Ok(()) } |
|
||||||
|
| ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
|
||||||
|
Remove the existing socket and run the code:
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
cargo run --bin server
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
it should hang. Perfect! The server is waiting for connections!
|
||||||
|
|
||||||
|
The client process wants to connect to an existing socket, read and write from it.
|
||||||
|
|
||||||
|
Next to `server.rs`, create the `client.rs` file. The client will merely consist of a `UnixStream`:
|
||||||
|
|
||||||
|
| 1 2 3 4 5 6 7 8 9 10 11 12 | // src/bin/client.rs use std::os::unix::net::{UnixListener, UnixStream}; use anyhow::Context; fn main() -> anyhow::Result<()> { let socket\_path = "mysocket"; let mut unix\_stream = UnixStream::connect(socket\_path).context("Could not create stream")?; Ok(()) |
|
||||||
|
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
|
||||||
|
We need to import the `Read` and `Write` traits.
|
||||||
|
|
||||||
|
| 1 2 | // src/bin/client.rs use std::io::{Read, Write}; |
|
||||||
|
| --- | ------------------------------------------------ |
|
||||||
|
|
||||||
|
And now we can write onto the stream. Below the `unix_stream` declaration, add the write logic:
|
||||||
|
|
||||||
|
| 1 2 3 | unix\_stream .write(b"Hello?") // we write bytes, &\[u8\] .context("Failed at writing onto the unix stream")?; |
|
||||||
|
| ----- | ---------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
|
||||||
|
Be sure to import `Read` and `Write` in `server.rs`:
|
||||||
|
|
||||||
|
| 1 2 | // src/bin/server.rs use std::io::{Read, Write}; |
|
||||||
|
| --- | ------------------------------------------------ |
|
||||||
|
|
||||||
|
Now let’s fill the `handle_stream` function with ordinary read logic:
|
||||||
|
|
||||||
|
| 1 2 3 4 5 6 7 8 9 10 | // src/bin/server.rs fn handle\_stream(mut unix\_stream: UnixStream) -> anyhow::Result<()> { let mut message = String::new(); unix\_stream .read\_to\_string(&mut message) .context("Failed at reading the unix stream")?; println!("{}", message); Ok(()) } |
|
||||||
|
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
|
||||||
|
Make sure you have the server running in a terminal:
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
cargo run --bin server
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
And in a separate terminal, run the client:
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
cargo run --bin client
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
If all is well, the hello message should display on the server side.
|
||||||
|
|
||||||
|
Let’s answer something every time the server receives anything.
|
||||||
|
|
||||||
|
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | // src/bin/server.rs fn handle\_stream(mut unix\_stream: UnixStream) -> anyhow::Result<()> { let mut message = String::new(); unix\_stream .read\_to\_string(&mut message) .context("Failed at reading the unix stream")?; println!("We received this message: {}\\nReplying...", message); unix\_stream .write(b"I hear you!") .context("Failed at writing onto the unix stream")?; Ok(()) } |
|
||||||
|
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
|
||||||
|
Introducing the same reading logic we used on the server **will not work**. Why? After writing on a stream, we need to shut down the writing, if we want to read from it.
|
||||||
|
|
||||||
|
Let’s segregate the write and read logic into distinct functions. Oh, and we pass mutable references (`&mut`) of the unix stream to the function, because… Rust. Don’t worry about it.
|
||||||
|
|
||||||
|
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | // src/bin/client.rs use std::io::{Read, Write}; use std::os::unix::net::{UnixListener, UnixStream}; use anyhow::Context; fn main() -> anyhow::Result<()> { let socket\_path = "mysocket"; let mut unix\_stream = UnixStream::connect(socket\_path).context("Could not create stream")?; write\_request\_and\_shutdown(&mut unix\_stream)?; read\_from\_stream(&mut unix\_stream)?; Ok(()) } |
|
||||||
|
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||||
|
|
||||||
|
The `shutdown()` method takes a `Shutdown` enum we would otherwise use on TCP streams. Write below the main function:
|
||||||
|
|
||||||
|
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | fn write\_request\_and\_shutdown(unix\_stream: &mut UnixStream) -> anyhow::Result<()> { unix\_stream .write(b"Hello?") .context("Failed at writing onto the unix stream")?; println!("We sent a request"); println!("Shutting down writing on the stream, waiting for response..."); unix\_stream .shutdown(std::net::Shutdown::Write) .context("Could not shutdown writing on the stream")?; Ok(()) } |
|
||||||
|
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
|
||||||
|
The stream is now clean to be read from.
|
||||||
|
|
||||||
|
| 1 2 3 4 5 6 7 8 9 | fn read\_from\_stream(unix\_stream: &mut UnixStream) -> anyhow::Result<()> { let mut response = String::new(); unix\_stream .read\_to\_string(&mut response) .context("Failed at reading the unix stream")?; println!("We received this response: {}", response); Ok(()) } |
|
||||||
|
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
|
||||||
|
Have the server running in a terminal:
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
cargo run --bin server
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
And in a separate terminal, run the client:
|
||||||
|
|
||||||
|
```routeros
|
||||||
|
cargo run --bin client
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
If all is well,
|
||||||
|
|
||||||
|
* the hello message should display on the server side
|
||||||
|
* the “I hear you” response should display on the client side
|
||||||
|
|
||||||
|
You can run the client as many times as you want, since the server runs in a loop.
|
||||||
|
|
||||||
|
This tutorial comes with a [github repository](https://github.com/Keksoj/unix%5Fsockets%5Fbasics)that contains the above code.
|
||||||
|
|
||||||
|
Feel free to write an issue for any comment, criticism, or complaint you may have. Fork and do pull requests as you please.
|
||||||
|
|
||||||
|
This blog post is a sum-up of what I learned trying to understand unix sockets while working on Sōzu. A more elaborate version of the code is available[in this other repo](https://github.com/Keksoj/unix%5Fsocket%5Fbased%5Fserver%5Fclient), with additional features:
|
||||||
|
|
||||||
|
* a `UnixListener`\-wrapping library with a glorious `SocketBuilder` helper (permissions! blocking/nonblocking!)
|
||||||
|
* a `Message` module with serializable `Request` and `Response` structs. The Response has a status that is either `Ok`, `Error` or `Processing`
|
||||||
|
* a client loop that continues reading the stream as long as responses come with a `Processing` status, to stops only at `Ok` or `Error`
|
||||||
|
|
||||||
|
All this happened thanks to my employer, [Clever Cloud](https://clever-cloud.com/), who allows me to learn my job in the best possible conditions. Much gratitude.
|
||||||
|
|
@ -0,0 +1,50 @@
|
||||||
|
---
|
||||||
|
id: fafb2eae-5872-4b07-8e3e-a215337ff70f
|
||||||
|
title: |
|
||||||
|
Bullet Journal in 5 Minutes a Day (for busy people)
|
||||||
|
status: ARCHIVED
|
||||||
|
tags:
|
||||||
|
- read-later
|
||||||
|
date_added: 2024-05-10 23:57:58
|
||||||
|
url_omnivore: |
|
||||||
|
https://omnivore.app/me/https-www-youtube-com-watch-v-t-op-4-hr-l-sc-4-18f65cce78b
|
||||||
|
url_original: |
|
||||||
|
https://www.youtube.com/watch?v=T_Op4hrLSc4
|
||||||
|
---
|
||||||
|
|
||||||
|
# Bullet Journal in 5 Minutes a Day (for Busy people)
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
KISS
|
||||||
|
|
||||||
|
In the morning:
|
||||||
|
|
||||||
|
- Write down what needs to be done in the day in a simple bullet list, ~5 actions
|
||||||
|
- Check previous day log
|
||||||
|
|
||||||
|
Daily logging:
|
||||||
|
|
||||||
|
- When switching contexts, write down what you did and what are you gonna do next
|
||||||
|
- When the action is completed, cross it off
|
||||||
|
- Single sentences
|
||||||
|
- Only write down what's noteworthy
|
||||||
|
|
||||||
|
Types of bullets:
|
||||||
|
|
||||||
|
- ` • `: actions
|
||||||
|
- ` - `: Notes, ideas & thoughts
|
||||||
|
- ` = `: Moods, mental & physical feelings
|
||||||
|
- ` ˚ `: Events
|
||||||
|
|
||||||
|
Reflection:
|
||||||
|
|
||||||
|
- Update the bullets
|
||||||
|
- Decided what to do with the uncompleted actions
|
||||||
|
- Write down final thoughts
|
||||||
|
|
||||||
|
## Original
|
||||||
|
|
||||||
|
[Bullet Journal in 5 Minutes a Day (for busy people)](https://www.youtube.com/watch?v=T%5FOp4hrLSc4)
|
||||||
|
|
||||||
|
By [Bullet Journal](https://www.youtube.com/@bulletjournal)
|
||||||
Loading…
Add table
Add a link
Reference in a new issue