formatting

This commit is contained in:
Robin Appelman 2025-05-26 18:42:12 +02:00
commit 2197181ab4
7 changed files with 132 additions and 126 deletions

View file

@ -6,8 +6,9 @@ Analysis tool for Nextcloud logs files
## Quickstart ## Quickstart
Download the binary from the [releases page](https://github.com/icewind1991/logsmash/releases), place it somewhere in Download the binary from the
your `$PATH` and make it executable. [releases page](https://github.com/icewind1991/logsmash/releases), place it
somewhere in your `$PATH` and make it executable.
```bash ```bash
logsmash ./logfile.log logsmash ./logfile.log
@ -15,34 +16,37 @@ logsmash ./logfile.log
## Log files ## Log files
Logsmash supports both loading plain log files, compressed log files (`.gz`, `.bz2`, `.xz` or `.zst`), or archives Logsmash supports both loading plain log files, compressed log files (`.gz`,
containing log files (`.zip` or (compressed) `.tar`). `.bz2`, `.xz` or `.zst`), or archives containing log files (`.zip` or
(compressed) `.tar`).
### Date formats ### Date formats
Since not all log files use the same date format, logsmash tries to parse each data with a number of different log Since not all log files use the same date format, logsmash tries to parse each
formats. data with a number of different log formats.
If the log file you're opening is using an unsupported log format, you can specify a custom date format with the If the log file you're opening is using an unsupported log format, you can
`--date-format` option. specify a custom date format with the `--date-format` option.
The data format can either be in [the strftime format](https://man7.org/linux/man-pages/man3/strftime.3.html) or The data format can either be in
in [the time crate format (version 2)](https://time-rs.github.io/book/api/format-description.html). [the strftime format](https://man7.org/linux/man-pages/man3/strftime.3.html) or
in
[the time crate format (version 2)](https://time-rs.github.io/book/api/format-description.html).
For example: `[day].[month].[year] - [hour]:[minute]:[second]`. For example: `[day].[month].[year] - [hour]:[minute]:[second]`.
## Log sources ## Log sources
Logsmash is built around matching log line to their source, either a call to a logging function or an exception being Logsmash is built around matching log line to their source, either a call to a
thrown. logging function or an exception being thrown. Many log lines do not include
Many log lines do not include information about their source to logsmash attempts to find the source of the log by information about their source to logsmash attempts to find the source of the
comparing log by comparing the logged message against a list of known log sources included
the logged message against a list of known log sources included in the binary. in the binary.
Since multiple log sources can create similar log lines, some lines will match to multiple log sources, in those cases Since multiple log sources can create similar log lines, some lines will match
logsmash to multiple log sources, in those cases logsmash will show all the matched
will show all the matched sources as a single item. sources as a single item. Log items that cannot be matched to any known source
Log items that cannot be matched to any known source are grouped together are grouped together
## UI overview ## UI overview
@ -50,14 +54,15 @@ Logsmash contains the following UI pages:
### Overview ### Overview
Shows a list of all matched sources with their source location, the number of log items matched to the source, and a Shows a list of all matched sources with their source location, the number of
graph of log occurrence over time. log items matched to the source, and a graph of log occurrence over time.
Selecting an item will bring you to the source overview for the log source. Selecting an item will bring you to the source overview for the log source.
### Source overview ### Source overview
Shows a list of distinct log lines (lines with the same message, level and context). Shows a list of distinct log lines (lines with the same message, level and
context).
Selecting an item will bring you to the log list Selecting an item will bring you to the log list
@ -69,22 +74,23 @@ Selecting an item will bring you to the log page
### Log page ### Log page
Shows the message and accompanying metadata for the log line. For log lines with exceptions, the exception backtrace is Shows the message and accompanying metadata for the log line. For log lines with
shown. exceptions, the exception backtrace is shown.
### Log occurrence graph ### Log occurrence graph
Lists that contain grouped log lines come with a graph that show how often the group of log lines occurred over time as Lists that contain grouped log lines come with a graph that show how often the
a histogram. group of log lines occurred over time as a histogram. Each line in the list will
Each line in the list will show a small version of the graph, and a larger version of the graph for the selected item is show a small version of the graph, and a larger version of the graph for the
shown above the list. selected item is shown above the list.
## Keyboard controls ## Keyboard controls
- All lists are navigated with the arrow keys, `PgUp`/`PgDown` and `Home`/`End`. - All lists are navigated with the arrow keys, `PgUp`/`PgDown` and `Home`/`End`.
- Items in lists can be selected with `Enter` or the right arrow key. - Items in lists can be selected with `Enter` or the right arrow key.
- You can return to the previous page with `Esc` or the left arrow key. - You can return to the previous page with `Esc` or the left arrow key.
- Single log items can be copied as json with `c` (requires [OSC 52 support](https://github.com/ojroques/vim-oscyank)) - Single log items can be copied as json with `c` (requires
[OSC 52 support](https://github.com/ojroques/vim-oscyank))
## Supported data ## Supported data
@ -119,14 +125,16 @@ Currently, the program can match against data from the following sources:
## Building ## Building
The preferred way of building is trough [`nix`](https://nixos.org/download/#download-nix), this ensures that all baked The preferred way of building is trough
[`nix`](https://nixos.org/download/#download-nix), this ensures that all baked
data is automatically up-to-date. data is automatically up-to-date.
`nix build .#logsmash` `nix build .#logsmash`
#### Updating baked data #### Updating baked data
Note that this is only needed when building with cargo, building with nix automatically uses the latest data. Note that this is only needed when building with cargo, building with nix
automatically uses the latest data.
```bash ```bash
rm -r data/src/data rm -r data/src/data

View file

@ -10,21 +10,22 @@
inputs.flakelight.follows = "flakelight"; inputs.flakelight.follows = "flakelight";
}; };
}; };
outputs = { mill-scale, ... }: mill-scale ./. { outputs = {mill-scale, ...}:
crossTargets = [ mill-scale ./. {
"x86_64-unknown-linux-musl" crossTargets = [
"x86_64-pc-windows-gnu" "x86_64-unknown-linux-musl"
]; "x86_64-pc-windows-gnu"
packageOpts = { extracted-logs-rust, ... }: { ];
preBuild = '' packageOpts = {extracted-logs-rust, ...}: {
rm -r -f data/src/data preBuild = ''
cp -r ${extracted-logs-rust} data/src/data rm -r -f data/src/data
''; cp -r ${extracted-logs-rust} data/src/data
'';
};
withOverlays = import ./nix/overlay.nix;
packages = {
logsmash = {logsmash}: logsmash;
extracted-logs-rust = {extracted-logs-rust}: extracted-logs-rust;
};
}; };
withOverlays = import ./nix/overlay.nix;
packages = {
logsmash = { logsmash }: logsmash;
extracted-logs-rust = { extracted-logs-rust }: extracted-logs-rust;
};
};
} }

View file

@ -1,9 +1,8 @@
{ packages {
, lib packages,
, writeTextDir lib,
, writeTextDir,
}: }: let
let
inherit (builtins) head attrNames concatStringsSep replaceStrings; inherit (builtins) head attrNames concatStringsSep replaceStrings;
inherit (lib.attrsets) mapAttrsToList; inherit (lib.attrsets) mapAttrsToList;
inherit (lib.lists) flatten last; inherit (lib.lists) flatten last;
@ -35,4 +34,4 @@ let
} }
''; '';
in in
writeTextDir "mod.rs" code writeTextDir "mod.rs" code

View file

@ -1,34 +1,36 @@
{ stdenvNoCC {
, logging-extractor stdenvNoCC,
, fetchzip logging-extractor,
, name fetchzip,
, version name,
, url version,
, major url,
, sha256 major,
, mode ? "json" sha256,
, mode ? "json",
}: }: let
let ext =
ext = if mode == "rust" then "rs" else "json"; if mode == "rust"
cleanedMajor = builtins.replaceStrings [ "." ] [ "_" ] major; then "rs"
else "json";
cleanedMajor = builtins.replaceStrings ["."] ["_"] major;
in in
stdenvNoCC.mkDerivation rec { stdenvNoCC.mkDerivation rec {
pname = "extractor-logs-${name}-${cleanedMajor}"; pname = "extractor-logs-${name}-${cleanedMajor}";
inherit version; inherit version;
src = fetchzip { src = fetchzip {
inherit url sha256; inherit url sha256;
}; };
nativeBuildInputs = [ logging-extractor ]; nativeBuildInputs = [logging-extractor];
buildPhase = '' buildPhase = ''
logging-extractor . ${mode} > logs.${ext} logging-extractor . ${mode} > logs.${ext}
''; '';
installPhase = '' installPhase = ''
mkdir -p $out mkdir -p $out
cp logs.* $out/${name}_${cleanedMajor}.${ext} cp logs.* $out/${name}_${cleanedMajor}.${ext}
''; '';
} }

View file

@ -1,19 +1,18 @@
{ stdenv {
, rustPlatform stdenv,
, lib rustPlatform,
, lib,
}: }: let
let
inherit (lib.sources) sourceByRegex; inherit (lib.sources) sourceByRegex;
src = sourceByRegex ../logging-extractor [ "Cargo.*" "(src|tests|test-data)(/.*)?" ]; src = sourceByRegex ../logging-extractor ["Cargo.*" "(src|tests|test-data)(/.*)?"];
in in
rustPlatform.buildRustPackage rec { rustPlatform.buildRustPackage rec {
pname = "logging-extractor"; pname = "logging-extractor";
version = "0.1.0"; version = "0.1.0";
inherit src; inherit src;
cargoLock = { cargoLock = {
lockFile = ../logging-extractor/Cargo.lock; lockFile = ../logging-extractor/Cargo.lock;
}; };
} }

View file

@ -1,29 +1,28 @@
{ stdenv {
, makeRustPlatform stdenv,
, rust-bin makeRustPlatform,
, lib rust-bin,
, extracted-logs-rust lib,
, extracted-logs-rust,
}: }: let
let
inherit (lib) sourceByRegex readFile; inherit (lib) sourceByRegex readFile;
src = sourceByRegex ../. [ "Cargo.*" "(src|data)(/.*)?" ]; src = sourceByRegex ../. ["Cargo.*" "(src|data)(/.*)?"];
rustPlatform = makeRustPlatform { rustPlatform = makeRustPlatform {
cargo = rust-bin.stable.latest.default; cargo = rust-bin.stable.latest.default;
rustc = rust-bin.stable.latest.default; rustc = rust-bin.stable.latest.default;
}; };
version = (fromTOML (readFile ../Cargo.toml)).package.version; version = (fromTOML (readFile ../Cargo.toml)).package.version;
in in
rustPlatform.buildRustPackage rec { rustPlatform.buildRustPackage rec {
pname = "logsmash"; pname = "logsmash";
inherit version src; inherit version src;
preBuild = '' preBuild = ''
rm -r data/src/data rm -r data/src/data
cp -r ${extracted-logs-rust} data/src/data cp -r ${extracted-logs-rust} data/src/data
''; '';
cargoLock = { cargoLock = {
lockFile = ../Cargo.lock; lockFile = ../Cargo.lock;
}; };
} }

View file

@ -1,5 +1,4 @@
final: prev: final: prev: let
let
inherit (builtins) mapAttrs attrValues map tail; inherit (builtins) mapAttrs attrValues map tail;
inherit (prev) symlinkJoin; inherit (prev) symlinkJoin;
inherit (prev.lib) importJSON; inherit (prev.lib) importJSON;
@ -16,19 +15,18 @@ let
inherit (data) url sha256 version; inherit (data) url sha256 version;
inherit name major mode; inherit name major mode;
})); }));
in in {
{ logging-extractor = final.callPackage ./logging-extractor.nix {};
logging-extractor = final.callPackage ./logging-extractor.nix { };
extracted-logs-parts = mapAttrs (loggingFor "json") packages; extracted-logs-parts = mapAttrs (loggingFor "json") packages;
extracted-logs-parts-rust = mapAttrs (loggingFor "rust") packages; extracted-logs-parts-rust = mapAttrs (loggingFor "rust") packages;
extracted-logs-rust-mod = final.callPackage ./data-mod.nix { inherit packages; }; extracted-logs-rust-mod = final.callPackage ./data-mod.nix {inherit packages;};
extracted-logs = symlinkJoin { extracted-logs = symlinkJoin {
name = "extracted-logs"; name = "extracted-logs";
paths = flatten (map attrValues (attrValues final.extracted-logs-parts)); paths = flatten (map attrValues (attrValues final.extracted-logs-parts));
}; };
extracted-logs-rust = symlinkJoin { extracted-logs-rust = symlinkJoin {
name = "extracted-logs"; name = "extracted-logs";
paths = (flatten (map attrValues (attrValues final.extracted-logs-parts-rust))) ++ [ final.extracted-logs-rust-mod ]; paths = (flatten (map attrValues (attrValues final.extracted-logs-parts-rust))) ++ [final.extracted-logs-rust-mod];
}; };
logsmash = final.callPackage ./logsmash.nix { }; logsmash = final.callPackage ./logsmash.nix {};
} }