1
0
mirror of https://github.com/chylex/Discord-History-Tracker.git synced 2025-08-17 19:31:42 +02:00

30 Commits

Author SHA1 Message Date
b660af4be0 WIP 2023-12-23 19:59:09 +01:00
3d9d6a454a Remove unnecessary ASP.NET features 2023-12-23 13:47:33 +01:00
ee39780928 Rewrite token authorization checks in integrated server 2023-12-23 13:47:31 +01:00
7b58f973a0 Disable ASP.NET logging and use custom logging for request duration 2023-12-23 11:19:54 +01:00
93fe018343 Add -console argument to show a console on Windows 2023-12-23 08:46:43 +01:00
4f5e27f651 Release v39.1 2023-12-22 16:31:11 +01:00
cbf81ec95a Fix missing JSON source generator when parsing integrated server requests 2023-12-22 16:31:11 +01:00
8a80cb8c20 Show progress dialog when upgrading database schema 2023-12-22 16:18:03 +01:00
865deb356a Fix progress dialog not propagating exceptions from its task 2023-12-22 14:47:55 +01:00
069ab97196 Disable reflection-based JSON serialization 2023-12-22 05:54:24 +01:00
caab038eaa Use source generators for JSON serialization everywhere 2023-12-22 05:24:28 +01:00
fb837374fc Enable single file compression and disable unnecessary .NET features 2023-12-22 02:30:24 +01:00
65d935cca1 Use compiled bindings in Avalonia XAML 2023-12-21 08:55:56 +01:00
6e64c86d7a Optimize viewer JSON export using source generators 2023-12-21 08:29:07 +01:00
8aeb590bb3 Release v39.0 2023-12-21 05:18:01 +01:00
8dc1adc9f0 Fix viewer converting underscores to italics even when not at the end of a word 2023-12-21 05:09:55 +01:00
ddf70b02e7 Fix not storing channel topic and position if the channel is in a category
Closes #232
2023-12-21 04:49:19 +01:00
ef59fd992e Reword "Do Nothing" option in autoscroll settings to improve clarity 2023-12-21 04:27:24 +01:00
d044627fac Reduce delay in tracker for responding to new messages 2023-12-21 04:27:24 +01:00
a624745602 Force light theme in Avalonia 2023-12-21 04:24:12 +01:00
6da3c185e5 Update Avalonia to 11.0.6 2023-12-21 04:24:01 +01:00
d4d14cab97 Update Rider project configuration files 2023-12-21 00:55:19 +01:00
095c9a061a Update attachment downloads for time-limited Discord CDN URLs
Closes #223
2023-12-21 00:27:08 +01:00
d01f9ed218 Rewrite detection of newly loaded messages 2023-12-20 13:19:11 +01:00
dd6f121059 Fix not seeing messages after a Discord update
Closes #240
2023-12-20 10:51:14 +01:00
8bba33d815 Update SQLite provider to 8.0.0 (SQLite version 3.41.2) 2023-12-20 10:42:17 +01:00
9eab8ac92a Update README section on distribution 2023-11-24 13:47:53 +01:00
fe588686fc Clean up project configuration and build scripts & fix CI builds 2023-11-23 21:45:54 +01:00
7392987165 Remove unused argument in tracker script
Closes #235
2023-11-23 05:22:33 +01:00
492dddb35d Load full tracking script from the integrated server to remove need for minification 2023-11-21 07:08:26 +01:00
140 changed files with 1478 additions and 42341 deletions

View File

@@ -16,7 +16,6 @@ Fork the repository and clone it to your computer (if you've never used git, you
Folder organization:
* `app/` contains a Visual Studio solution for the desktop app
* `lib/` contains utilities required to build the project
* `web/` contains source code of the [official website](https://dht.chylex.com), which can be used as a template when making your own website
To start editing source code for the desktop app, install the [.NET 8 SDK](https://dotnet.microsoft.com/en-us/download/dotnet/8.0), and then open `app/DiscordHistoryTracker.sln` in [Visual Studio](https://visualstudio.microsoft.com/downloads/) or [Rider](https://www.jetbrains.com/rider/).
@@ -29,22 +28,13 @@ To build a `Release` version of the desktop app, follow the instructions for you
#### Release Windows (64-bit)
1. Install [Python 3](https://www.python.org/downloads), and ensure the `python` executable is in your `PATH`
2. Install [Powershell 5](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows) or newer (on Windows 10, the included version of Powershell should be enough)
The `lib/` folder contains an installation of [Node](https://nodejs.org/en) and [uglify-js](https://www.npmjs.com/package/uglify-js), which are used to minify the tracking script. This installation will only work on 64-bit Windows; building on 32-bit Windows is not supported, but you can try.
1. Install [Powershell 5](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows) or newer (on Windows 10, the included version of Powershell should be enough)
Run the `app/build.bat` script, and read the [Distribution](#distribution) section below.
#### Release Other Operating Systems
1. Install [Python 3](https://www.python.org/downloads), and ensure the `python` executable exists and launches Python 3
- On Debian and derivatives, you can install `python-is-python3`
- On other distributions, you can create a link manually, for ex. `ln -s /usr/bin/python3 /usr/bin/python`
- If you don't want `python` to mean Python 3, then edit `Desktop.csproj` and change `python` to `python3`
2. Install [Node + npm](https://nodejs.org/en)
3. Install [uglify-js](https://www.npmjs.com/package/uglify-js) globally (`npm install -g uglify-js`)
4. Install the `zip` package from your repository
1. Install the `zip` package from your repository
Run the `app/build.sh` script, and read the [Distribution](#distribution) section below.
@@ -52,4 +42,4 @@ Run the `app/build.sh` script, and read the [Distribution](#distribution) sectio
The mentioned build scripts will prepare `Release` builds ready for distribution. Once the script finishes, the `app/bin` folder will contain self-contained executables for each major operating system, and a portable version that works on all other systems but requires .NET 8 to be installed.
Note that when building on Windows, the generated `.zip` files for Linux and Mac will not have correct file permissions, so it will not be possible to run them by double-clicking `DiscordHistoryTracker`. I tried using Python to re-create the archives with correct file permissions, but found that Linux `zip` tools could not see them. The only working solution is building the Windows + portable version on Windows, and Linux + Mac version on Linux.
Note that when building on Windows, the generated `.zip` files for Linux and Mac will not have correct file permissions, so it will not be possible to run them by double-clicking the executable. Since .NET 8 fixed several issues with publishing Windows executables on Linux, I recommend using Linux to build the app for all operating systems.

View File

@@ -29,9 +29,6 @@
<H2CodeStyleSettings version="6">
<option name="USE_GENERIC_STYLE" value="true" />
</H2CodeStyleSettings>
<H2CodeStyleSettings version="6">
<option name="USE_GENERIC_STYLE" value="true" />
</H2CodeStyleSettings>
<HSQLCodeStyleSettings version="6">
<option name="USE_GENERIC_STYLE" value="true" />
</HSQLCodeStyleSettings>

View File

@@ -1,8 +1,8 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Desktop" type="DotNetProject" factoryName=".NET Project">
<option name="EXE_PATH" value="$PROJECT_DIR$/Desktop/bin/Debug/net5.0/DiscordHistoryTracker.exe" />
<option name="EXE_PATH" value="$PROJECT_DIR$/.artifacts/bin/Desktop/debug/DiscordHistoryTracker.exe" />
<option name="PROGRAM_PARAMETERS" value="" />
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$/Desktop/bin/Debug/net5.0" />
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$/.artifacts/bin/Desktop/debug" />
<option name="PASS_PARENT_ENVS" value="1" />
<option name="USE_EXTERNAL_CONSOLE" value="0" />
<option name="USE_MONO" value="0" />
@@ -12,7 +12,7 @@
<option name="PROJECT_ARGUMENTS_TRACKING" value="1" />
<option name="PROJECT_WORKING_DIRECTORY_TRACKING" value="1" />
<option name="PROJECT_KIND" value="DotNetCore" />
<option name="PROJECT_TFM" value="net5.0" />
<option name="PROJECT_TFM" value="net8.0" />
<method v="2">
<option name="Build" />
</method>

View File

@@ -1,23 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Minify" type="PythonConfigurationType" factoryName="Python">
<module name="rider.module" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="PARENT_ENVS" value="true" />
<envs>
<env name="PYTHONUNBUFFERED" value="1" />
</envs>
<option name="SDK_HOME" value="" />
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$" />
<option name="IS_MODULE_SDK" value="false" />
<option name="ADD_CONTENT_ROOTS" value="false" />
<option name="ADD_SOURCE_ROOTS" value="false" />
<option name="SCRIPT_NAME" value="$PROJECT_DIR$/minify.py" />
<option name="PARAMETERS" value="" />
<option name="SHOW_COMMAND_LINE" value="false" />
<option name="EMULATE_TERMINAL" value="false" />
<option name="MODULE_MODE" value="false" />
<option name="REDIRECT_INPUT" value="false" />
<option name="INPUT_FILE" value="" />
<method v="2" />
</configuration>
</component>

View File

@@ -2,7 +2,8 @@
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:common="clr-namespace:DHT.Desktop.Common"
xmlns:system="clr-namespace:System;assembly=System.Runtime"
x:Class="DHT.Desktop.App">
x:Class="DHT.Desktop.App"
RequestedThemeVariant="Light">
<Application.Styles>

View File

@@ -1,4 +1,3 @@
using System;
using Avalonia;
using Avalonia.Controls.ApplicationLifetimes;
using Avalonia.Markup.Xaml;
@@ -13,7 +12,7 @@ sealed class App : Application {
public override void OnFrameworkInitializationCompleted() {
if (ApplicationLifetime is IClassicDesktopStyleApplicationLifetime desktop) {
desktop.MainWindow = new MainWindow(new Arguments(desktop.Args ?? Array.Empty<string>()));
desktop.MainWindow = new MainWindow(Program.Arguments);
}
base.OnFrameworkInitializationCompleted();

View File

@@ -5,26 +5,33 @@ namespace DHT.Desktop;
sealed class Arguments {
private static readonly Log Log = Log.ForType<Arguments>();
private const int FirstArgument = 1;
public static Arguments Empty => new(Array.Empty<string>());
public bool Console { get; }
public string? DatabaseFile { get; }
public ushort? ServerPort { get; }
public string? ServerToken { get; }
public Arguments(string[] args) {
for (int i = 0; i < args.Length; i++) {
for (int i = FirstArgument; i < args.Length; i++) {
string key = args[i];
switch (key) {
case "-debug":
Log.IsDebugEnabled = true;
continue;
case "-console":
Console = true;
continue;
}
string value;
if (i == 0 && !key.StartsWith('-')) {
if (i == FirstArgument && !key.StartsWith('-')) {
value = key;
key = "-db";
}

View File

@@ -1,9 +1,11 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using Avalonia.Controls;
using Avalonia.Platform.Storage;
using Avalonia.Threading;
using DHT.Desktop.Dialogs.File;
using DHT.Desktop.Dialogs.Message;
using DHT.Server.Database;
@@ -41,11 +43,16 @@ static class DatabaseGui {
});
}
public static async Task<IDatabaseFile?> TryOpenOrCreateDatabaseFromPath(string path, Window window, Func<Task<bool>> checkCanUpgradeDatabase) {
public static async Task<IDatabaseFile?> TryOpenOrCreateDatabaseFromPath(string path, Window window, ISchemaUpgradeCallbacks schemaUpgradeCallbacks) {
var prevSynchronizationContext = SynchronizationContext.Current;
SynchronizationContext.SetSynchronizationContext(new AvaloniaSynchronizationContext());
var taskScheduler = TaskScheduler.FromCurrentSynchronizationContext();
SynchronizationContext.SetSynchronizationContext(prevSynchronizationContext);
IDatabaseFile? file = null;
try {
file = await SqliteDatabaseFile.OpenOrCreate(path, checkCanUpgradeDatabase);
file = await SqliteDatabaseFile.OpenOrCreate(path, schemaUpgradeCallbacks, taskScheduler);
} catch (InvalidDatabaseVersionException ex) {
await Dialog.ShowOk(window, "Database Error", "Database '" + Path.GetFileName(path) + "' appears to be corrupted (invalid version: " + ex.Version + ").");
} catch (DatabaseTooNewException ex) {

View File

@@ -1,36 +1,33 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<RootNamespace>DHT.Desktop</RootNamespace>
<AssemblyName>DiscordHistoryTracker</AssemblyName>
<PackageId>DiscordHistoryTracker</PackageId>
</PropertyGroup>
<PropertyGroup>
<OutputType>WinExe</OutputType>
<Nullable>enable</Nullable>
<AssemblyName>DiscordHistoryTracker</AssemblyName>
<RootNamespace>DHT.Desktop</RootNamespace>
<PackageId>DiscordHistoryTracker</PackageId>
<Authors>chylex</Authors>
<Company>DiscordHistoryTracker</Company>
<Product>DiscordHistoryTracker</Product>
<ApplicationIcon>./Resources/icon.ico</ApplicationIcon>
<AvaloniaUseCompiledBindingsByDefault>true</AvaloniaUseCompiledBindingsByDefault>
<CheckForOverflowUnderflow>true</CheckForOverflowUnderflow>
<SatelliteResourceLanguages>en</SatelliteResourceLanguages>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<DebugSymbols>true</DebugSymbols>
<DebugType>none</DebugType>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Avalonia" Version="11.0.5" />
<PackageReference Include="Avalonia.Controls.DataGrid" Version="11.0.5" />
<PackageReference Include="Avalonia.Controls.ItemsRepeater" Version="11.0.5" />
<PackageReference Include="Avalonia.Desktop" Version="11.0.5" />
<PackageReference Include="Avalonia.Diagnostics" Version="11.0.5" Condition=" '$(Configuration)' == 'Debug' " />
<PackageReference Include="Avalonia.Fonts.Inter" Version="11.0.5" />
<PackageReference Include="Avalonia.Themes.Fluent" Version="11.0.5" />
<PackageReference Include="Avalonia" Version="11.0.6" />
<PackageReference Include="Avalonia.Controls.DataGrid" Version="11.0.6" />
<PackageReference Include="Avalonia.Controls.ItemsRepeater" Version="11.0.6" />
<PackageReference Include="Avalonia.Desktop" Version="11.0.6" />
<PackageReference Include="Avalonia.Diagnostics" Version="11.0.6" Condition=" '$(Configuration)' == 'Debug' " />
<PackageReference Include="Avalonia.Fonts.Inter" Version="11.0.6" />
<PackageReference Include="Avalonia.Themes.Fluent" Version="11.0.6" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\Server\Server.csproj" />
</ItemGroup>
<ItemGroup>
<Compile Include="..\Version.cs" Link="Version.cs" />
<Compile Update="Dialogs\TextBox\TextBoxDialog.axaml.cs">
@@ -38,22 +35,11 @@
<SubType>Code</SubType>
</Compile>
</ItemGroup>
<ItemGroup>
<AvaloniaResource Include="Resources/icon.ico" />
<EmbeddedResource Include="../Resources/Tracker/bootstrap.js">
<LogicalName>Tracker\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
</EmbeddedResource>
<EmbeddedResource Include="../Resources/Tracker/scripts/**" Condition=" '$(Configuration)' == 'Debug' ">
<LogicalName>Tracker\scripts\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/scripts/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
</EmbeddedResource>
<EmbeddedResource Include="../Resources/Tracker/styles/**">
<LogicalName>Tracker\styles\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/styles/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
<EmbeddedResource Include="Resources/tracker-loader.js">
<LogicalName>tracker-loader.js</LogicalName>
</EmbeddedResource>
<EmbeddedResource Include="../Resources/Viewer/**">
<LogicalName>Viewer\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
@@ -61,19 +47,5 @@
<Visible>false</Visible>
</EmbeddedResource>
</ItemGroup>
<Target Name="MinifyResources" BeforeTargets="PrepareForBuild" Condition=" '$(Configuration)' == 'Release' ">
<PropertyGroup>
<MinifiedResourceDir>$(ProjectDir)bin/.res/scripts</MinifiedResourceDir>
</PropertyGroup>
<ItemGroup>
<UpToDateCheckInput Include="$(ProjectDir)../Resources/Tracker/scripts/**" Visible="false" />
<EmbeddedResource Include="$(MinifiedResourceDir)/discord.js" LogicalName="Tracker\scripts\discord.js" Visible="false" />
<EmbeddedResource Include="$(MinifiedResourceDir)/dom.js" LogicalName="Tracker\scripts\dom.js" Visible="false" />
<EmbeddedResource Include="$(MinifiedResourceDir)/gui.js" LogicalName="Tracker\scripts\gui.js" Visible="false" />
<EmbeddedResource Include="$(MinifiedResourceDir)/settings.js" LogicalName="Tracker\scripts\settings.js" Visible="false" />
<EmbeddedResource Include="$(MinifiedResourceDir)/state.js" LogicalName="Tracker\scripts\state.js" Visible="false" />
</ItemGroup>
<RemoveDir Directories="$(ProjectDir)bin/.res/scripts" />
<Exec Command="python $(ProjectDir)../Resources/minify.py" WorkingDirectory="$(ProjectDir)../Resources" IgnoreExitCode="false" />
</Target>
</Project>

View File

@@ -5,6 +5,7 @@
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.CheckBox"
mc:Ignorable="d" d:DesignWidth="500"
x:Class="DHT.Desktop.Dialogs.CheckBox.CheckBoxDialog"
x:DataType="namespace:CheckBoxDialogModel"
Title="{Binding Title}"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Width="500" SizeToContent="Height" CanResize="False"

View File

@@ -5,6 +5,7 @@
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.Message"
mc:Ignorable="d" d:DesignWidth="500"
x:Class="DHT.Desktop.Dialogs.Message.MessageDialog"
x:DataType="namespace:MessageDialogModel"
Title="{Binding Title}"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Width="500" SizeToContent="Height" CanResize="False"

View File

@@ -4,4 +4,5 @@ namespace DHT.Desktop.Dialogs.Progress;
interface IProgressCallback {
Task Update(string message, int finishedItems, int totalItems);
Task Hide();
}

View File

@@ -5,6 +5,7 @@
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.Progress"
mc:Ignorable="d" d:DesignWidth="500"
x:Class="DHT.Desktop.Dialogs.Progress.ProgressDialog"
x:DataType="namespace:ProgressDialogModel"
Title="{Binding Title}"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Opened="OnOpened"
@@ -31,12 +32,18 @@
</Style>
</Window.Styles>
<StackPanel Margin="20">
<DockPanel>
<TextBlock DockPanel.Dock="Right" Text="{Binding Items}" Classes="items" />
<TextBlock DockPanel.Dock="Left" Text="{Binding Message}" />
</DockPanel>
<ProgressBar Value="{Binding Progress}" />
</StackPanel>
<ItemsRepeater ItemsSource="{Binding Items}" Margin="0 10">
<ItemsRepeater.ItemTemplate>
<DataTemplate>
<StackPanel Margin="20 10" IsHitTestVisible="{Binding IsVisible}" Opacity="{Binding Opacity}">
<DockPanel>
<TextBlock DockPanel.Dock="Right" Text="{Binding Items}" Classes="items" />
<TextBlock DockPanel.Dock="Left" Text="{Binding Message}" />
</DockPanel>
<ProgressBar Value="{Binding Progress}" />
</StackPanel>
</DataTemplate>
</ItemsRepeater.ItemTemplate>
</ItemsRepeater>
</Window>

View File

@@ -8,6 +8,7 @@ namespace DHT.Desktop.Dialogs.Progress;
[SuppressMessage("ReSharper", "MemberCanBeInternal")]
public sealed partial class ProgressDialog : Window {
private bool isFinished = false;
private Task progressTask = Task.CompletedTask;
public ProgressDialog() {
InitializeComponent();
@@ -15,7 +16,8 @@ public sealed partial class ProgressDialog : Window {
public void OnOpened(object? sender, EventArgs e) {
if (DataContext is ProgressDialogModel model) {
Task.Run(model.StartTask).ContinueWith(OnFinished, TaskScheduler.FromCurrentSynchronizationContext());
progressTask = Task.Run(model.StartTask);
progressTask.ContinueWith(OnFinished, TaskScheduler.FromCurrentSynchronizationContext());
}
}
@@ -27,4 +29,9 @@ public sealed partial class ProgressDialog : Window {
isFinished = true;
Close();
}
public async Task ShowProgressDialog(Window owner) {
await ShowDialog(owner);
await progressTask;
}
}

View File

@@ -1,4 +1,6 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Avalonia.Threading;
using DHT.Desktop.Common;
@@ -9,57 +11,43 @@ namespace DHT.Desktop.Dialogs.Progress;
sealed class ProgressDialogModel : BaseModel {
public string Title { get; init; } = "";
private string message = "";
public string Message {
get => message;
private set => Change(ref message, value);
}
private string items = "";
public string Items {
get => items;
private set => Change(ref items, value);
}
private int progress = 0;
public int Progress {
get => progress;
private set => Change(ref progress, value);
}
public IReadOnlyList<ProgressItem> Items { get; } = Array.Empty<ProgressItem>();
private readonly TaskRunner? task;
[Obsolete("Designer")]
public ProgressDialogModel() {}
public ProgressDialogModel(TaskRunner task) {
public ProgressDialogModel(TaskRunner task, int progressItems = 1) {
this.Items = Enumerable.Range(0, progressItems).Select(static _ => new ProgressItem()).ToArray();
this.task = task;
}
internal async Task StartTask() {
if (task != null) {
await task(new Callback(this));
await task(Items.Select(static item => new Callback(item)).ToArray());
}
}
public delegate Task TaskRunner(IProgressCallback callback);
public delegate Task TaskRunner(IReadOnlyList<IProgressCallback> callbacks);
private sealed class Callback : IProgressCallback {
private readonly ProgressDialogModel model;
private readonly ProgressItem item;
public Callback(ProgressDialogModel model) {
this.model = model;
public Callback(ProgressItem item) {
this.item = item;
}
async Task IProgressCallback.Update(string message, int finishedItems, int totalItems) {
public async Task Update(string message, int finishedItems, int totalItems) {
await Dispatcher.UIThread.InvokeAsync(() => {
model.Message = message;
model.Items = finishedItems.Format() + " / " + totalItems.Format();
model.Progress = 100 * finishedItems / totalItems;
item.Message = message;
item.Items = totalItems == 0 ? string.Empty : finishedItems.Format() + " / " + totalItems.Format();
item.Progress = totalItems == 0 ? 0 : 100 * finishedItems / totalItems;
});
}
public Task Hide() {
return Update(string.Empty, 0, 0);
}
}
}

View File

@@ -0,0 +1,41 @@
using DHT.Utils.Models;
namespace DHT.Desktop.Dialogs.Progress;
sealed class ProgressItem : BaseModel {
private bool isVisible = false;
public bool IsVisible {
get => isVisible;
private set {
Change(ref isVisible, value);
OnPropertyChanged(nameof(Opacity));
}
}
public double Opacity => IsVisible ? 1.0 : 0.0;
private string message = "";
public string Message {
get => message;
set {
Change(ref message, value);
IsVisible = !string.IsNullOrEmpty(value);
}
}
private string items = "";
public string Items {
get => items;
set => Change(ref items, value);
}
private int progress = 0;
public int Progress {
get => progress;
set => Change(ref progress, value);
}
}

View File

@@ -5,6 +5,7 @@
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.TextBox"
mc:Ignorable="d" d:DesignWidth="500"
x:Class="DHT.Desktop.Dialogs.TextBox.TextBoxDialog"
x:DataType="namespace:TextBoxDialogModel"
Title="{Binding Title}"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Width="500" SizeToContent="Height" CanResize="False"

View File

@@ -1,9 +1,9 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.IO;
using System.Runtime.InteropServices;
using System.Text.Json;
using System.Text.Json.Nodes;
using System.Threading.Tasks;
using DHT.Utils.Logging;
using static System.Environment.SpecialFolder;
@@ -47,12 +47,12 @@ static class DiscordAppSettings {
}
}
private static bool AreDevToolsEnabled(Dictionary<string, object?> json) {
return json.TryGetValue(JsonKeyDevTools, out var value) && value is JsonElement { ValueKind: JsonValueKind.True };
private static bool AreDevToolsEnabled(JsonObject json) {
return json.TryGetPropertyValue(JsonKeyDevTools, out var node) && node?.GetValueKind() == JsonValueKind.True;
}
public static async Task<SettingsJsonResult> ConfigureDevTools(bool enable) {
Dictionary<string, object?> json;
JsonObject json;
try {
json = await ReadSettingsJson();
@@ -109,13 +109,13 @@ static class DiscordAppSettings {
return SettingsJsonResult.Success;
}
private static async Task<Dictionary<string, object?>> ReadSettingsJson() {
private static async Task<JsonObject> ReadSettingsJson() {
await using var stream = new FileStream(JsonFilePath, FileMode.Open, FileAccess.Read, FileShare.Read);
return await JsonSerializer.DeserializeAsync<Dictionary<string, object?>?>(stream) ?? throw new JsonException();
return await JsonSerializer.DeserializeAsync(stream, DiscordAppSettingsJsonContext.Default.JsonObject) ?? throw new JsonException();
}
private static async Task WriteSettingsJson(Dictionary<string, object?> json) {
private static async Task WriteSettingsJson(JsonObject json) {
await using var stream = new FileStream(JsonFilePath, FileMode.Truncate, FileAccess.Write, FileShare.None);
await JsonSerializer.SerializeAsync(stream, json, new JsonSerializerOptions { WriteIndented = true });
await JsonSerializer.SerializeAsync(stream, json, DiscordAppSettingsJsonContext.Default.JsonObject);
}
}

View File

@@ -0,0 +1,8 @@
using System.Text.Json.Nodes;
using System.Text.Json.Serialization;
namespace DHT.Desktop.Discord;
[JsonSourceGenerationOptions(GenerationMode = JsonSourceGenerationMode.Default, WriteIndented = true)]
[JsonSerializable(typeof(JsonObject))]
sealed partial class DiscordAppSettingsJsonContext : JsonSerializerContext {}

View File

@@ -5,6 +5,7 @@
xmlns:main="clr-namespace:DHT.Desktop.Main"
mc:Ignorable="d" d:DesignWidth="480" d:DesignHeight="295"
x:Class="DHT.Desktop.Main.AboutWindow"
x:DataType="main:AboutWindowModel"
Title="About Discord History Tracker"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Width="480" Height="295" CanResize="False"

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d"
x:Class="DHT.Desktop.Main.Controls.AttachmentFilterPanel">
x:Class="DHT.Desktop.Main.Controls.AttachmentFilterPanel"
x:DataType="controls:AttachmentFilterPanelModel">
<Design.DataContext>
<controls:AttachmentFilterPanelModel />

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d"
x:Class="DHT.Desktop.Main.Controls.MessageFilterPanel">
x:Class="DHT.Desktop.Main.Controls.MessageFilterPanel"
x:DataType="controls:MessageFilterPanelModel">
<Design.DataContext>
<controls:MessageFilterPanelModel />

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d"
x:Class="DHT.Desktop.Main.Controls.ServerConfigurationPanel">
x:Class="DHT.Desktop.Main.Controls.ServerConfigurationPanel"
x:DataType="controls:ServerConfigurationPanelModel">
<Design.DataContext>
<controls:ServerConfigurationPanelModel />

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d"
x:Class="DHT.Desktop.Main.Controls.StatusBar">
x:Class="DHT.Desktop.Main.Controls.StatusBar"
x:DataType="controls:StatusBarModel">
<Design.DataContext>
<controls:StatusBarModel />

View File

@@ -5,6 +5,7 @@
xmlns:main="clr-namespace:DHT.Desktop.Main"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.MainWindow"
x:DataType="main:MainWindowModel"
Title="{Binding Title}"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Width="800" Height="500"

View File

@@ -5,7 +5,8 @@
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.AdvancedPage">
x:Class="DHT.Desktop.Main.Pages.AdvancedPage"
x:DataType="pages:AdvancedPageModel">
<Design.DataContext>
<pages:AdvancedPageModel />

View File

@@ -5,7 +5,8 @@
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.AttachmentsPage">
x:Class="DHT.Desktop.Main.Pages.AttachmentsPage"
x:DataType="pages:AttachmentsPageModel">
<Design.DataContext>
<pages:AttachmentsPageModel />
@@ -35,7 +36,7 @@
<TextBlock Text="{Binding DownloadMessage}" Margin="10 0 0 0" VerticalAlignment="Center" DockPanel.Dock="Left" />
<ProgressBar Value="{Binding DownloadProgress}" IsVisible="{Binding IsDownloading}" Margin="15 0" VerticalAlignment="Center" DockPanel.Dock="Right" />
</DockPanel>
<controls:AttachmentFilterPanel DataContext="{Binding FilterModel}" IsEnabled="{Binding !DataContext.IsDownloading, RelativeSource={RelativeSource AncestorType=UserControl}}" />
<controls:AttachmentFilterPanel DataContext="{Binding FilterModel}" IsEnabled="{Binding !IsDownloading, RelativeSource={RelativeSource AncestorType=pages:AttachmentsPageModel}}" />
<StackPanel Orientation="Vertical" Spacing="12">
<Expander Header="Download Status" IsExpanded="True">
<DataGrid ItemsSource="{Binding StatisticsRows}" AutoGenerateColumns="False" CanUserReorderColumns="False" CanUserResizeColumns="False" CanUserSortColumns="False" IsReadOnly="True">

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.DatabasePage">
x:Class="DHT.Desktop.Main.Pages.DatabasePage"
x:DataType="pages:DatabasePageModel">
<Design.DataContext>
<pages:DatabasePageModel />

View File

@@ -17,6 +17,7 @@ using DHT.Desktop.Dialogs.TextBox;
using DHT.Server.Data;
using DHT.Server.Database;
using DHT.Server.Database.Import;
using DHT.Server.Database.Sqlite;
using DHT.Utils.Logging;
using DHT.Utils.Models;
@@ -77,31 +78,18 @@ sealed class DatabasePageModel : BaseModel {
}
ProgressDialog progressDialog = new ProgressDialog();
progressDialog.DataContext = new ProgressDialogModel(async callback => await MergeWithDatabaseFromPaths(Db, paths, progressDialog, callback)) {
progressDialog.DataContext = new ProgressDialogModel(async callbacks => await MergeWithDatabaseFromPaths(Db, paths, progressDialog, callbacks[0])) {
Title = "Database Merge"
};
await progressDialog.ShowDialog(window);
await progressDialog.ShowProgressDialog(window);
}
private static async Task MergeWithDatabaseFromPaths(IDatabaseFile target, string[] paths, ProgressDialog dialog, IProgressCallback callback) {
int total = paths.Length;
DialogResult.YesNo? upgradeResult = null;
async Task<bool> CheckCanUpgradeDatabase() {
upgradeResult ??= total > 1
? await DatabaseGui.ShowCanUpgradeMultipleDatabaseDialog(dialog)
: await DatabaseGui.ShowCanUpgradeDatabaseDialog(dialog);
return DialogResult.YesNo.Yes == upgradeResult;
}
var schemaUpgradeCallbacks = new SchemaUpgradeCallbacks(dialog, paths.Length);
await PerformImport(target, paths, dialog, callback, "Database Merge", "Database Error", "database file", async path => {
SynchronizationContext? prevSyncContext = SynchronizationContext.Current;
SynchronizationContext.SetSynchronizationContext(new AvaloniaSynchronizationContext());
IDatabaseFile? db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, dialog, CheckCanUpgradeDatabase);
SynchronizationContext.SetSynchronizationContext(prevSyncContext);
IDatabaseFile? db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, dialog, schemaUpgradeCallbacks);
if (db == null) {
return false;
@@ -116,6 +104,41 @@ sealed class DatabasePageModel : BaseModel {
});
}
private sealed class SchemaUpgradeCallbacks : ISchemaUpgradeCallbacks {
private readonly ProgressDialog dialog;
private readonly int total;
private bool? decision;
public SchemaUpgradeCallbacks(ProgressDialog dialog, int total) {
this.total = total;
this.dialog = dialog;
}
public async Task<bool> CanUpgrade() {
return decision ??= (total > 1
? await DatabaseGui.ShowCanUpgradeMultipleDatabaseDialog(dialog)
: await DatabaseGui.ShowCanUpgradeDatabaseDialog(dialog)) == DialogResult.YesNo.Yes;
}
public Task Start(int versionSteps, Func<ISchemaUpgradeCallbacks.IProgressReporter, Task> doUpgrade) {
return doUpgrade(new NullReporter());
}
private sealed class NullReporter : ISchemaUpgradeCallbacks.IProgressReporter {
public Task NextVersion() {
return Task.CompletedTask;
}
public Task MainWork(string message, int finishedItems, int totalItems) {
return Task.CompletedTask;
}
public Task SubWork(string message, int finishedItems, int totalItems) {
return Task.CompletedTask;
}
}
}
public async void ImportLegacyArchive() {
var paths = await window.StorageProvider.OpenFiles(new FilePickerOpenOptions {
Title = "Open Legacy DHT Archive",
@@ -128,11 +151,11 @@ sealed class DatabasePageModel : BaseModel {
}
ProgressDialog progressDialog = new ProgressDialog();
progressDialog.DataContext = new ProgressDialogModel(async callback => await ImportLegacyArchiveFromPaths(Db, paths, progressDialog, callback)) {
progressDialog.DataContext = new ProgressDialogModel(async callbacks => await ImportLegacyArchiveFromPaths(Db, paths, progressDialog, callbacks[0])) {
Title = "Legacy Archive Import"
};
await progressDialog.ShowDialog(window);
await progressDialog.ShowProgressDialog(window);
}
private static async Task ImportLegacyArchiveFromPaths(IDatabaseFile target, string[] paths, ProgressDialog dialog, IProgressCallback callback) {

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.DebugPage">
x:Class="DHT.Desktop.Main.Pages.DebugPage"
x:DataType="pages:DebugPageModel">
<Design.DataContext>
<pages:DebugPageModel />

View File

@@ -45,12 +45,12 @@ namespace DHT.Desktop.Main.Pages {
}
ProgressDialog progressDialog = new ProgressDialog {
DataContext = new ProgressDialogModel(async callback => await GenerateRandomData(channels, users, messages, callback)) {
DataContext = new ProgressDialogModel(async callbacks => await GenerateRandomData(channels, users, messages, callbacks[0])) {
Title = "Generating Random Data"
}
};
await progressDialog.ShowDialog(window);
await progressDialog.ShowProgressDialog(window);
}
private const int BatchSize = 500;

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.TrackingPage">
x:Class="DHT.Desktop.Main.Pages.TrackingPage"
x:DataType="pages:TrackingPageModel">
<Design.DataContext>
<pages:TrackingPageModel />

View File

@@ -52,14 +52,10 @@ sealed class TrackingPageModel : BaseModel {
OnPropertyChanged(nameof(IsToggleAppDevToolsButtonEnabled));
}
}
public async Task<bool> OnClickCopyTrackingScript() {
string bootstrap = await Resources.ReadTextAsync("Tracker/bootstrap.js");
string script = bootstrap.Replace("= 0; /*[PORT]*/", "= " + ServerManager.Port + ";")
.Replace("/*[TOKEN]*/", HttpUtility.JavaScriptStringEncode(ServerManager.Token))
.Replace("/*[IMPORTS]*/", await Resources.ReadJoinedAsync("Tracker/scripts/", '\n'))
.Replace("/*[CSS-CONTROLLER]*/", await Resources.ReadTextAsync("Tracker/styles/controller.css"))
.Replace("/*[CSS-SETTINGS]*/", await Resources.ReadTextAsync("Tracker/styles/settings.css"));
string url = $"http://127.0.0.1:{ServerManager.Port}/get-tracking-script?token={HttpUtility.UrlEncode(ServerManager.Token)}";
string script = (await Resources.ReadTextAsync("tracker-loader.js")).Trim().Replace("{url}", url);
var clipboard = window.Clipboard;
if (clipboard == null) {

View File

@@ -5,7 +5,8 @@
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.ViewerPage">
x:Class="DHT.Desktop.Main.Pages.ViewerPage"
x:DataType="pages:ViewerPageModel">
<Design.DataContext>
<pages:ViewerPageModel />

View File

@@ -35,7 +35,7 @@ sealed class ViewerPageModel : BaseModel, IDisposable {
set => Change(ref hasFilters, value);
}
private MessageFilterPanelModel FilterModel { get; }
public MessageFilterPanelModel FilterModel { get; }
private readonly Window window;
private readonly IDatabaseFile db;
@@ -65,6 +65,8 @@ sealed class ViewerPageModel : BaseModel, IDisposable {
string indexFile = await Resources.ReadTextAsync("Viewer/index.html");
string viewerTemplate = indexFile.Replace("/*[JS]*/", await Resources.ReadJoinedAsync("Viewer/scripts/", '\n'))
.Replace("/*[CSS]*/", await Resources.ReadJoinedAsync("Viewer/styles/", '\n'));
viewerTemplate = strategy.ProcessViewerTemplate(viewerTemplate);
int viewerArchiveTagStart = viewerTemplate.IndexOf(ArchiveTag);
int viewerArchiveTagEnd = viewerArchiveTagStart + ArchiveTag.Length;

View File

@@ -5,7 +5,8 @@
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
xmlns:screens="clr-namespace:DHT.Desktop.Main.Screens"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Screens.MainContentScreen">
x:Class="DHT.Desktop.Main.Screens.MainContentScreen"
x:DataType="screens:MainContentScreenModel">
<Design.DataContext>
<screens:MainContentScreenModel />

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:screens="clr-namespace:DHT.Desktop.Main.Screens"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Screens.WelcomeScreen">
x:Class="DHT.Desktop.Main.Screens.WelcomeScreen"
x:DataType="screens:WelcomeScreenModel">
<Design.DataContext>
<screens:WelcomeScreenModel />

View File

@@ -1,10 +1,13 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
using Avalonia.Controls;
using DHT.Desktop.Common;
using DHT.Desktop.Dialogs.Message;
using DHT.Desktop.Dialogs.Progress;
using DHT.Server.Database;
using DHT.Server.Database.Sqlite;
using DHT.Utils.Models;
namespace DHT.Desktop.Main.Screens;
@@ -39,14 +42,71 @@ sealed class WelcomeScreenModel : BaseModel, IDisposable {
}
dbFilePath = path;
Db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, window, CheckCanUpgradeDatabase);
Db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, window, new SchemaUpgradeCallbacks(window));
OnPropertyChanged(nameof(Db));
OnPropertyChanged(nameof(HasDatabase));
}
private async Task<bool> CheckCanUpgradeDatabase() {
return DialogResult.YesNo.Yes == await DatabaseGui.ShowCanUpgradeDatabaseDialog(window);
private sealed class SchemaUpgradeCallbacks : ISchemaUpgradeCallbacks {
private readonly Window window;
public SchemaUpgradeCallbacks(Window window) {
this.window = window;
}
public async Task<bool> CanUpgrade() {
return DialogResult.YesNo.Yes == await DatabaseGui.ShowCanUpgradeDatabaseDialog(window);
}
public async Task Start(int versionSteps, Func<ISchemaUpgradeCallbacks.IProgressReporter, Task> doUpgrade) {
async Task StartUpgrade(IReadOnlyList<IProgressCallback> callbacks) {
var reporter = new ProgressReporter(versionSteps, callbacks);
await reporter.NextVersion();
await Task.Delay(TimeSpan.FromMilliseconds(800));
await doUpgrade(reporter);
await Task.Delay(TimeSpan.FromMilliseconds(600));
}
await new ProgressDialog {
DataContext = new ProgressDialogModel(StartUpgrade, progressItems: 3) {
Title = "Upgrading Database"
}
}.ShowProgressDialog(window);
}
private sealed class ProgressReporter : ISchemaUpgradeCallbacks.IProgressReporter {
private readonly IReadOnlyList<IProgressCallback> callbacks;
private readonly int versionSteps;
private int versionProgress = 0;
public ProgressReporter(int versionSteps, IReadOnlyList<IProgressCallback> callbacks) {
this.callbacks = callbacks;
this.versionSteps = versionSteps;
}
public async Task NextVersion() {
await callbacks[0].Update("Upgrading schema version...", versionProgress++, versionSteps);
await HideChildren(0);
}
public async Task MainWork(string message, int finishedItems, int totalItems) {
await callbacks[1].Update(message, finishedItems, totalItems);
await HideChildren(1);
}
public async Task SubWork(string message, int finishedItems, int totalItems) {
await callbacks[2].Update(message, finishedItems, totalItems);
await HideChildren(2);
}
private async Task HideChildren(int parentIndex) {
for (int i = parentIndex + 1; i < callbacks.Count; i++) {
await callbacks[i].Hide();
}
}
}
}
public void CloseDatabase() {

View File

@@ -1,6 +1,8 @@
using System.Globalization;
using System;
using System.Globalization;
using System.Reflection;
using Avalonia;
using DHT.Utils.Logging;
using DHT.Utils.Resources;
namespace DHT.Desktop;
@@ -9,6 +11,7 @@ static class Program {
public static string Version { get; }
public static CultureInfo Culture { get; }
public static ResourceLoader Resources { get; }
public static Arguments Arguments { get; }
static Program() {
var assembly = Assembly.GetExecutingAssembly();
@@ -25,10 +28,21 @@ static class Program {
CultureInfo.DefaultThreadCurrentUICulture = CultureInfo.InvariantCulture;
Resources = new ResourceLoader(assembly);
Arguments = new Arguments(Environment.GetCommandLineArgs());
}
public static void Main(string[] args) {
BuildAvaloniaApp().StartWithClassicDesktopLifetime(args);
if (Arguments.Console && OperatingSystem.IsWindows()) {
WindowsConsole.AllocConsole();
}
try {
BuildAvaloniaApp().StartWithClassicDesktopLifetime(args);
} finally {
if (Arguments.Console && OperatingSystem.IsWindows()) {
WindowsConsole.FreeConsole();
}
}
}
private static AppBuilder BuildAvaloniaApp() {

View File

@@ -0,0 +1 @@
fetch("{url}").then(r => r.ok ? (r.headers.get("X-DHT") === "1" ? r.text() : Promise.reject("Invalid response")) : Promise.reject(r.status + " " + r.statusText)).then(s => eval(s)).catch(e => alert("Could not load tracking script:\n" + e));

49
app/Directory.Build.props Normal file
View File

@@ -0,0 +1,49 @@
<Project>
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<LangVersion>11</LangVersion>
<Nullable>enable</Nullable>
</PropertyGroup>
<PropertyGroup>
<Authors>chylex</Authors>
<Company>DiscordHistoryTracker</Company>
<Product>DiscordHistoryTracker</Product>
</PropertyGroup>
<PropertyGroup>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
</PropertyGroup>
<PropertyGroup>
<SuppressTrimAnalysisWarnings>false</SuppressTrimAnalysisWarnings>
<PublishTrimmed>true</PublishTrimmed>
<TrimMode>partial</TrimMode>
<EnableUnsafeBinaryFormatterSerialization>false</EnableUnsafeBinaryFormatterSerialization>
<EnableUnsafeUTF7Encoding>false</EnableUnsafeUTF7Encoding>
<EventSourceSupport>false</EventSourceSupport>
<HttpActivityPropagationSupport>false</HttpActivityPropagationSupport>
<JsonSerializerIsReflectionEnabledByDefault>false</JsonSerializerIsReflectionEnabledByDefault>
</PropertyGroup>
<PropertyGroup>
<PublishSingleFile>true</PublishSingleFile>
<PublishReadyToRun>false</PublishReadyToRun>
<EnableCompressionInSingleFile>true</EnableCompressionInSingleFile>
<IncludeNativeLibrariesForSelfExtract>true</IncludeNativeLibrariesForSelfExtract>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<DebugSymbols>false</DebugSymbols>
<DebugType>none</DebugType>
</PropertyGroup>
<PropertyGroup>
<UseArtifactsOutput>true</UseArtifactsOutput>
<ArtifactsPath>$(MSBuildThisFileDirectory).artifacts</ArtifactsPath>
</PropertyGroup>
</Project>

View File

@@ -1,8 +0,0 @@
<Project>
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<LangVersion>11</LangVersion>
</PropertyGroup>
</Project>

View File

@@ -64,9 +64,11 @@
let action = null;
if (!DISCORD.hasMoreMessages()) {
console.debug("[DHT] Reached first message.");
action = SETTINGS.afterFirstMsg;
}
if (isNoAction(action) && !anyNewMessages) {
console.debug("[DHT] No new messages.");
action = SETTINGS.afterSavedMsg;
}
@@ -121,7 +123,7 @@
onTrackingContinued(false);
}
else {
const anyNewMessages = await STATE.addDiscordMessages(info.id, messages);
const anyNewMessages = await STATE.addDiscordMessages(messages);
onTrackingContinued(anyNewMessages);
}
} catch (e) {
@@ -156,3 +158,4 @@
GUI.showSettings();
}
})();
/*[DEBUGGER]*/

View File

@@ -1,5 +1,23 @@
// noinspection JSUnresolvedVariable
// noinspection LocalVariableNamingConventionJS
class DISCORD {
// https://discord.com/developers/docs/resources/channel#channel-object-channel-types
static CHANNEL_TYPE = {
DM: 1,
GROUP_DM: 3,
ANNOUNCEMENT_THREAD: 10,
PUBLIC_THREAD: 11,
PRIVATE_THREAD: 12
};
// https://discord.com/developers/docs/resources/channel#message-object-message-types
static MESSAGE_TYPE = {
DEFAULT: 0,
REPLY: 19,
THREAD_STARTER: 21
};
static getMessageOuterElement() {
return DOM.queryReactClass("messagesWrapper");
}
@@ -28,46 +46,11 @@ class DISCORD {
* Calls the provided function with a list of messages whenever the currently loaded messages change.
*/
static setupMessageCallback(callback) {
let skipsLeft = 0;
let waitForCleanup = false;
const previousMessages = new Set();
const timer = window.setInterval(() => {
if (skipsLeft > 0) {
--skipsLeft;
return;
}
const view = this.getMessageOuterElement();
if (!view) {
skipsLeft = 2;
return;
}
const anyMessage = DOM.queryReactClass("message", this.getMessageOuterElement());
const messageCount = anyMessage ? anyMessage.parentElement.children.length : 0;
if (messageCount > 300) {
if (waitForCleanup) {
return;
}
skipsLeft = 3;
waitForCleanup = true;
window.setTimeout(() => {
const view = this.getMessageScrollerElement();
// noinspection JSUnusedGlobalSymbols
view.scrollTop = view.scrollHeight / 2;
}, 1);
}
else {
waitForCleanup = false;
}
const messages = this.getMessages();
const hasChanged = messages.some(message => !previousMessages.has(message.id)) || !this.hasMoreMessages();
const onMessageElementsChanged = function() {
const messages = DISCORD.getMessages();
const hasChanged = messages.some(message => !previousMessages.has(message.id)) || !DISCORD.hasMoreMessages();
if (!hasChanged) {
return;
@@ -79,24 +62,74 @@ class DISCORD {
}
callback(messages);
}, 200);
};
window.DHT_ON_UNLOAD.push(() => window.clearInterval(timer));
let debounceTimer;
/**
* Do not trigger the callback too often due to autoscrolling.
*/
const onMessageElementsChangedLater = function() {
window.clearTimeout(debounceTimer);
debounceTimer = window.setTimeout(onMessageElementsChanged, 100);
};
const observer = new MutationObserver(function () {
onMessageElementsChangedLater();
});
let skipsLeft = 0;
let observedElement = null;
const observerTimer = window.setInterval(() => {
if (skipsLeft > 0) {
--skipsLeft;
return;
}
const view = this.getMessageOuterElement();
if (!view) {
skipsLeft = 1;
return;
}
if (observedElement !== null && observedElement.isConnected) {
return;
}
observedElement = view.querySelector("[data-list-id='chat-messages']");
if (observedElement) {
console.debug("[DHT] Observed message container.");
observer.observe(observedElement, { childList: true });
onMessageElementsChangedLater();
}
}, 400);
window.DHT_ON_UNLOAD.push(() => {
observer.disconnect();
observedElement = null;
window.clearInterval(observerTimer);
});
}
/**
* Returns the property object of a message element.
* @returns { null | { message: DiscordMessage, channel: Object } }
* Returns the message from a message element.
* @returns { null | DiscordMessage } }
*/
static getMessageElementProps(ele) {
static getMessageFromElement(ele) {
const props = DOM.getReactProps(ele);
if (props.children && props.children.length) {
for (let i = 3; i < props.children.length; i++) {
const childProps = props.children[i].props;
if (props && Array.isArray(props.children)) {
for (const child of props.children) {
if (!(child instanceof Object)) {
continue;
}
if (childProps && "message" in childProps && "channel" in childProps) {
return childProps;
const childProps = child.props;
if (childProps instanceof Object && "message" in childProps) {
return childProps.message;
}
}
}
@@ -113,10 +146,10 @@ class DISCORD {
for (const ele of this.getMessageElements()) {
try {
const props = this.getMessageElementProps(ele);
const message = this.getMessageFromElement(ele);
if (props != null) {
messages.push(props.message);
if (message != null) {
messages.push(message);
}
} catch (e) {
console.error("[DHT] Error extracing message data, skipping it.", e, ele, DOM.tryGetReactProps(ele));
@@ -137,7 +170,7 @@ class DISCORD {
*/
static getSelectedChannel() {
try {
let obj;
let obj = null;
try {
for (const child of DOM.getReactProps(DOM.queryReactClass("chatContent")).children) {
@@ -148,15 +181,6 @@ class DISCORD {
}
} catch (e) {
console.error("[DHT] Error retrieving selected channel from 'chatContent' element.", e);
for (const ele of this.getMessageElements()) {
const props = this.getMessageElementProps(ele);
if (props != null) {
obj = props.channel;
break;
}
}
}
if (!obj || typeof obj.id !== "string") {
@@ -185,8 +209,8 @@ class DISCORD {
// https://discord.com/developers/docs/resources/channel#channel-object-channel-types
switch (obj.type) {
case 1: type = "DM"; break;
case 3: type = "GROUP"; break;
case DISCORD.CHANNEL_TYPE.DM: type = "DM"; break;
case DISCORD.CHANNEL_TYPE.GROUP_DM: type = "GROUP"; break;
default: return null;
}
@@ -224,7 +248,7 @@ class DISCORD {
}
};
if (obj.parent_id) {
if (obj.type === DISCORD.CHANNEL_TYPE.ANNOUNCEMENT_THREAD || obj.type === DISCORD.CHANNEL_TYPE.PUBLIC_THREAD || obj.type === DISCORD.CHANNEL_TYPE.PRIVATE_THREAD) {
channel["extra"]["parent"] = obj.parent_id;
}
else {

View File

@@ -86,12 +86,12 @@ const GUI = (function() {
<label><input id='dht-cfg-autoscroll' type='checkbox'> Autoscroll</label><br>
<br>
<label>After reaching the first message in channel...</label><br>
${radio("afm", "nothing", "Do Nothing")}
${radio("afm", "nothing", "Continue Tracking")}
${radio("afm", "pause", "Pause Tracking")}
${radio("afm", "switch", "Switch to Next Channel")}
<br>
<label>After reaching a previously saved message...</label><br>
${radio("asm", "nothing", "Do Nothing")}
${radio("asm", "nothing", "Continue Tracking")}
${radio("asm", "pause", "Pause Tracking")}
${radio("asm", "switch", "Switch to Next Channel")}
<p id='dht-cfg-note'>It is recommended to disable link and image previews to avoid putting unnecessary strain on your browser.</p>`;

View File

@@ -174,12 +174,10 @@ const STATE = (function() {
},
/**
* @param {String} channelId
* @param {DiscordMessage[]} discordMessageArray
*/
async addDiscordMessages(channelId, discordMessageArray) {
// https://discord.com/developers/docs/resources/channel#message-object-message-types
discordMessageArray = discordMessageArray.filter(msg => (msg.type === 0 || msg.type === 19 || msg.type === 21) && msg.state === "SENT");
async addDiscordMessages(discordMessageArray) {
discordMessageArray = discordMessageArray.filter(msg => (msg.type === DISCORD.MESSAGE_TYPE.DEFAULT || msg.type === DISCORD.MESSAGE_TYPE.REPLY || msg.type === DISCORD.MESSAGE_TYPE.THREAD_STARTER) && msg.state === "SENT");
if (discordMessageArray.length === 0) {
return false;

View File

@@ -6,6 +6,8 @@
<script type="text/javascript">
window.DHT_EMBEDDED = "/*[ARCHIVE]*/";
window.DHT_SERVER_URL = "/*[SERVER_URL]*/";
window.DHT_SERVER_TOKEN = "/*[SERVER_TOKEN]*/";
/*[JS]*/
</script>
<style>

View File

@@ -1,7 +1,8 @@
const DISCORD = (function() {
const regex = {
formatBold: /\*\*([\s\S]+?)\*\*(?!\*)/g,
formatItalic: /(.)?([_*])([\s\S]+?)\2(?!\2)/g,
formatItalic1: /\*([\s\S]+?)\*(?!\*)/g,
formatItalic2: /_([\s\S]+?)_(?!_)\b/g,
formatUnderline: /__([\s\S]+?)__(?!_)/g,
formatStrike: /~~([\s\S]+?)~~(?!~)/g,
formatCodeInline: /(`+)\s*([\s\S]*?[^`])\s*\1(?!`)/g,
@@ -48,7 +49,8 @@ const DISCORD = (function() {
.replace(regex.specialEscapedDouble, full => full.replace(/\\/g, "").replace(/(.)/g, escapeHtmlMatch))
.replace(regex.formatBold, "<b>$1</b>")
.replace(regex.formatUnderline, "<u>$1</u>")
.replace(regex.formatItalic, (full, pre, char, match) => pre === "\\" ? full : (pre || "") + "<i>" + match + "</i>")
.replace(regex.formatItalic1, "<i>$1</i>")
.replace(regex.formatItalic2, "<i>$1</i>")
.replace(regex.formatStrike, "<s>$1</s>");
}

View File

@@ -182,15 +182,32 @@ const STATE = (function() {
return null;
};
const getMessageList = function() {
const getMessageList = async function(abortSignal) {
if (!loadedMessages) {
return [];
}
const messages = getMessages(selectedChannel);
const startIndex = messagesPerPage * (root.getCurrentPage() - 1);
const slicedMessages = loadedMessages.slice(startIndex, !messagesPerPage ? undefined : startIndex + messagesPerPage);
return loadedMessages.slice(startIndex, !messagesPerPage ? undefined : startIndex + messagesPerPage).map(key => {
let messageTexts = null;
if (window.DHT_SERVER_URL !== null) {
const messageIds = new Set(slicedMessages);
for (const key of slicedMessages) {
const message = messages[key];
if ("r" in message) {
messageIds.add(message.r);
}
}
messageTexts = await getMessageTextsFromServer(messageIds, abortSignal);
}
return slicedMessages.map(key => {
/**
* @type {{}}
* @property {Number} u
@@ -216,6 +233,9 @@ const STATE = (function() {
if ("m" in message) {
obj["contents"] = message.m;
}
else if (messageTexts && key in messageTexts) {
obj["contents"] = messageTexts[key];
}
if ("e" in message) {
obj["embeds"] = message.e.map(embed => JSON.parse(embed));
@@ -230,15 +250,16 @@ const STATE = (function() {
}
if ("r" in message) {
const replyMessage = getMessageById(message.r);
const replyId = message.r;
const replyMessage = getMessageById(replyId);
const replyUser = replyMessage ? getUser(replyMessage.u) : null;
const replyAvatar = replyUser && replyUser.avatar ? { id: getUserId(replyMessage.u), path: replyUser.avatar } : null;
obj["reply"] = replyMessage ? {
"id": message.r,
"id": replyId,
"user": replyUser,
"avatar": replyAvatar,
"contents": replyMessage.m
"contents": messageTexts != null && replyId in messageTexts ? messageTexts[replyId] : replyMessage.m,
} : null;
}
@@ -250,9 +271,35 @@ const STATE = (function() {
});
};
const getMessageTextsFromServer = async function(messageIds, abortSignal) {
let idParams = "";
for (const messageId of messageIds) {
idParams += "id=" + encodeURIComponent(messageId) + "&";
}
const response = await fetch(DHT_SERVER_URL + "/get-messages?" + idParams + "token=" + encodeURIComponent(DHT_SERVER_TOKEN), {
method: "GET",
headers: {
"Content-Type": "application/json",
},
credentials: "omit",
redirect: "error",
signal: abortSignal
});
if (response.status === 200) {
return response.json();
}
else {
throw new Error("Server returned status " + response.status + " " + response.statusText);
}
};
let eventOnUsersRefreshed;
let eventOnChannelsRefreshed;
let eventOnMessagesRefreshed;
let messageLoaderAborter = null;
const triggerUsersRefreshed = function() {
eventOnUsersRefreshed && eventOnUsersRefreshed(getUserList());
@@ -263,7 +310,22 @@ const STATE = (function() {
};
const triggerMessagesRefreshed = function() {
eventOnMessagesRefreshed && eventOnMessagesRefreshed(getMessageList());
if (!eventOnMessagesRefreshed) {
return;
}
if (messageLoaderAborter != null) {
messageLoaderAborter.abort();
}
const aborter = new AbortController();
messageLoaderAborter = aborter;
getMessageList(aborter.signal).then(eventOnMessagesRefreshed).finally(() => {
if (messageLoaderAborter === aborter) {
messageLoaderAborter = null;
}
});
};
const getFilteredMessageKeys = function(channel) {

View File

@@ -1,27 +0,0 @@
#!/usr/bin/env python3
import glob
import os
import shutil
import sys
if os.name == "nt":
uglifyjs = os.path.abspath("../../lib/uglifyjs.cmd")
else:
uglifyjs = "uglifyjs"
if shutil.which(uglifyjs) is None:
print("Cannot find executable: {0}".format(uglifyjs))
sys.exit(1)
input_dir = os.path.abspath("./Tracker/scripts")
output_dir = os.path.abspath("../Desktop/bin/.res/scripts")
os.makedirs(output_dir, exist_ok=True)
for file in glob.glob(input_dir + "/*.js"):
name = os.path.basename(file)
print("Minifying {0}...".format(name))
os.system("{0} {1} -o {2}/{3}".format(uglifyjs, file, output_dir, name))
print("Done!")

View File

@@ -4,7 +4,8 @@ public readonly struct Attachment {
public ulong Id { get; internal init; }
public string Name { get; internal init; }
public string? Type { get; internal init; }
public string Url { get; internal init; }
public string NormalizedUrl { get; internal init; }
public string DownloadUrl { get; internal init; }
public ulong Size { get; internal init; }
public int? Width { get; internal init; }
public int? Height { get; internal init; }

View File

@@ -1,30 +1,33 @@
using System;
using System.Net;
using DHT.Server.Download;
namespace DHT.Server.Data;
public readonly struct Download {
internal static Download NewSuccess(string url, byte[] data) {
return new Download(url, DownloadStatus.Success, (ulong) Math.Max(data.LongLength, 0), data);
internal static Download NewSuccess(DownloadItem item, byte[] data) {
return new Download(item.NormalizedUrl, item.DownloadUrl, DownloadStatus.Success, (ulong) Math.Max(data.LongLength, 0), data);
}
internal static Download NewFailure(string url, HttpStatusCode? statusCode, ulong size) {
return new Download(url, statusCode.HasValue ? (DownloadStatus) (int) statusCode : DownloadStatus.GenericError, size);
internal static Download NewFailure(DownloadItem item, HttpStatusCode? statusCode, ulong size) {
return new Download(item.NormalizedUrl, item.DownloadUrl, statusCode.HasValue ? (DownloadStatus) (int) statusCode : DownloadStatus.GenericError, size);
}
public string Url { get; }
public string NormalizedUrl { get; }
public string DownloadUrl { get; }
public DownloadStatus Status { get; }
public ulong Size { get; }
public byte[]? Data { get; }
internal Download(string url, DownloadStatus status, ulong size, byte[]? data = null) {
Url = url;
internal Download(string normalizedUrl, string downloadUrl, DownloadStatus status, ulong size, byte[]? data = null) {
NormalizedUrl = normalizedUrl;
DownloadUrl = downloadUrl;
Status = status;
Size = size;
Data = data;
}
internal Download WithData(byte[] data) {
return new Download(Url, Status, Size, data);
return new Download(NormalizedUrl, DownloadUrl, Status, Size, data);
}
}

View File

@@ -44,7 +44,7 @@ public sealed class DummyDatabaseFile : IDatabaseFile {
return 0;
}
public List<Message> GetMessages(MessageFilter? filter = null) {
public List<Message> GetMessages(MessageFilter? filter = null, bool includeText = true) {
return new();
}

View File

@@ -0,0 +1,3 @@
namespace DHT.Server.Database.Export;
readonly record struct Snowflake(ulong Id);

View File

@@ -0,0 +1,23 @@
using System;
using System.Text.Json;
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Export;
sealed class SnowflakeJsonSerializer : JsonConverter<Snowflake> {
public override Snowflake Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) {
return new Snowflake(ulong.Parse(reader.GetString()!));
}
public override void Write(Utf8JsonWriter writer, Snowflake value, JsonSerializerOptions options) {
writer.WriteStringValue(value.Id.ToString());
}
public override Snowflake ReadAsPropertyName(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) {
return new Snowflake(ulong.Parse(reader.GetString()!));
}
public override void WriteAsPropertyName(Utf8JsonWriter writer, Snowflake value, JsonSerializerOptions options) {
writer.WritePropertyName(value.Id.ToString());
}
}

View File

@@ -3,5 +3,7 @@ using DHT.Server.Data;
namespace DHT.Server.Database.Export.Strategy;
public interface IViewerExportStrategy {
bool IncludeMessageText { get; }
string ProcessViewerTemplate(string template);
string GetAttachmentUrl(Attachment attachment);
}

View File

@@ -12,7 +12,14 @@ public sealed class LiveViewerExportStrategy : IViewerExportStrategy {
this.safeToken = WebUtility.UrlEncode(token);
}
public bool IncludeMessageText => false;
public string ProcessViewerTemplate(string template) {
return template.Replace("/*[SERVER_URL]*/", "http://127.0.0.1:" + safePort)
.Replace("/*[SERVER_TOKEN]*/", WebUtility.UrlEncode(safeToken));
}
public string GetAttachmentUrl(Attachment attachment) {
return "http://127.0.0.1:" + safePort + "/get-attachment/" + WebUtility.UrlEncode(attachment.Url) + "?token=" + safeToken;
return "http://127.0.0.1:" + safePort + "/get-attachment/" + WebUtility.UrlEncode(attachment.NormalizedUrl) + "?token=" + safeToken;
}
}

View File

@@ -7,7 +7,19 @@ public sealed class StandaloneViewerExportStrategy : IViewerExportStrategy {
private StandaloneViewerExportStrategy() {}
public bool IncludeMessageText => true;
public string ProcessViewerTemplate(string template) {
return template.Replace("\"/*[SERVER_URL]*/\"", "null")
.Replace("\"/*[SERVER_TOKEN]*/\"", "null");
}
public string GetAttachmentUrl(Attachment attachment) {
return attachment.Url;
// The normalized URL will not load files from Discord CDN once the time limit is enforced.
// The downloaded URL would work, but only for a limited time, so it is better for the links to not work
// rather than give users a false sense of security.
return attachment.NormalizedUrl;
}
}

View File

@@ -0,0 +1,93 @@
using System.Collections.Generic;
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Export;
sealed class ViewerJson {
public required JsonMeta Meta { get; init; }
public required Dictionary<Snowflake, Dictionary<Snowflake, JsonMessage>> Data { get; init; }
public sealed class JsonMeta {
public required Dictionary<Snowflake, JsonUser> Users { get; init; }
public required List<Snowflake> Userindex { get; init; }
public required List<JsonServer> Servers { get; init; }
public required Dictionary<Snowflake, JsonChannel> Channels { get; init; }
}
public sealed class JsonUser {
public required string Name { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Avatar { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Tag { get; init; }
}
public sealed class JsonServer {
public required string Name { get; init; }
public required string Type { get; init; }
}
public sealed class JsonChannel {
public required int Server { get; init; }
public required string Name { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Parent { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public int? Position { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Topic { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public bool? Nsfw { get; init; }
}
public sealed class JsonMessage {
public required int U { get; init; }
public required long T { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? M { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public long? Te { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? R { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public JsonMessageAttachment[]? A { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string[]? E { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public JsonMessageReaction[]? Re { get; init; }
}
public sealed class JsonMessageAttachment {
public required string Url { get; init; }
public required string Name { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public int? Width { get; set; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public int? Height { get; set; }
}
public sealed class JsonMessageReaction {
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Id { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? N { get; init; }
public required bool A { get; init; }
public required int C { get; init; }
}
}

View File

@@ -0,0 +1,11 @@
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Export;
[JsonSourceGenerationOptions(
Converters = new [] { typeof(SnowflakeJsonSerializer) },
PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase,
GenerationMode = JsonSourceGenerationMode.Default
)]
[JsonSerializable(typeof(ViewerJson))]
sealed partial class ViewerJsonContext : JsonSerializerContext {}

View File

@@ -21,7 +21,7 @@ public static class ViewerJsonExport {
var includedChannelIds = new HashSet<ulong>();
var includedServerIds = new HashSet<ulong>();
var includedMessages = db.GetMessages(filter);
var includedMessages = db.GetMessages(filter, strategy.IncludeMessageText);
var includedChannels = new List<Channel>();
foreach (var message in includedMessages) {
@@ -42,26 +42,28 @@ public static class ViewerJsonExport {
perf.Step("Collect database data");
var value = new {
meta = new { users, userindex, servers, channels },
data = GenerateMessageList(includedMessages, userIndices, strategy),
var value = new ViewerJson {
Meta = new ViewerJson.JsonMeta {
Users = users,
Userindex = userindex,
Servers = servers,
Channels = channels
},
Data = GenerateMessageList(includedMessages, userIndices, strategy)
};
perf.Step("Generate value object");
var opts = new JsonSerializerOptions();
opts.Converters.Add(new ViewerJsonSnowflakeSerializer());
await JsonSerializer.SerializeAsync(stream, value, opts);
await JsonSerializer.SerializeAsync(stream, value, ViewerJsonContext.Default.ViewerJson);
perf.Step("Serialize to JSON");
perf.End();
}
private static Dictionary<string, object> GenerateUserList(IDatabaseFile db, HashSet<ulong> userIds, out List<string> userindex, out Dictionary<ulong, object> userIndices) {
var users = new Dictionary<string, object>();
userindex = new List<string>();
userIndices = new Dictionary<ulong, object>();
private static Dictionary<Snowflake, ViewerJson.JsonUser> GenerateUserList(IDatabaseFile db, HashSet<ulong> userIds, out List<Snowflake> userindex, out Dictionary<ulong, int> userIndices) {
var users = new Dictionary<Snowflake, ViewerJson.JsonUser>();
userindex = new List<Snowflake>();
userIndices = new Dictionary<ulong, int>();
foreach (var user in db.GetAllUsers()) {
var id = user.Id;
@@ -69,30 +71,23 @@ public static class ViewerJsonExport {
continue;
}
var obj = new Dictionary<string, object> {
["name"] = user.Name
};
if (user.AvatarUrl != null) {
obj["avatar"] = user.AvatarUrl;
}
if (user.Discriminator != null) {
obj["tag"] = user.Discriminator;
}
var idStr = id.ToString();
var idSnowflake = new Snowflake(id);
userIndices[id] = users.Count;
userindex.Add(idStr);
users[idStr] = obj;
userindex.Add(idSnowflake);
users[idSnowflake] = new ViewerJson.JsonUser {
Name = user.Name,
Avatar = user.AvatarUrl,
Tag = user.Discriminator
};
}
return users;
}
private static List<object> GenerateServerList(IDatabaseFile db, HashSet<ulong> serverIds, out Dictionary<ulong, object> serverIndices) {
var servers = new List<object>();
serverIndices = new Dictionary<ulong, object>();
private static List<ViewerJson.JsonServer> GenerateServerList(IDatabaseFile db, HashSet<ulong> serverIds, out Dictionary<ulong, int> serverIndices) {
var servers = new List<ViewerJson.JsonServer>();
serverIndices = new Dictionary<ulong, int>();
foreach (var server in db.GetAllServers()) {
var id = server.Id;
@@ -101,113 +96,78 @@ public static class ViewerJsonExport {
}
serverIndices[id] = servers.Count;
servers.Add(new Dictionary<string, object> {
["name"] = server.Name,
["type"] = ServerTypes.ToJsonViewerString(server.Type),
servers.Add(new ViewerJson.JsonServer {
Name = server.Name,
Type = ServerTypes.ToJsonViewerString(server.Type)
});
}
return servers;
}
private static Dictionary<string, object> GenerateChannelList(List<Channel> includedChannels, Dictionary<ulong, object> serverIndices) {
var channels = new Dictionary<string, object>();
private static Dictionary<Snowflake, ViewerJson.JsonChannel> GenerateChannelList(List<Channel> includedChannels, Dictionary<ulong, int> serverIndices) {
var channels = new Dictionary<Snowflake, ViewerJson.JsonChannel>();
foreach (var channel in includedChannels) {
var obj = new Dictionary<string, object> {
["server"] = serverIndices[channel.Server],
["name"] = channel.Name,
var channelIdSnowflake = new Snowflake(channel.Id);
channels[channelIdSnowflake] = new ViewerJson.JsonChannel {
Server = serverIndices[channel.Server],
Name = channel.Name,
Parent = channel.ParentId?.ToString(),
Position = channel.Position,
Topic = channel.Topic,
Nsfw = channel.Nsfw
};
if (channel.ParentId != null) {
obj["parent"] = channel.ParentId;
}
if (channel.Position != null) {
obj["position"] = channel.Position;
}
if (channel.Topic != null) {
obj["topic"] = channel.Topic;
}
if (channel.Nsfw != null) {
obj["nsfw"] = channel.Nsfw;
}
channels[channel.Id.ToString()] = obj;
}
return channels;
}
private static Dictionary<string, Dictionary<string, object>> GenerateMessageList( List<Message> includedMessages, Dictionary<ulong, object> userIndices, IViewerExportStrategy strategy) {
var data = new Dictionary<string, Dictionary<string, object>>();
private static Dictionary<Snowflake, Dictionary<Snowflake, ViewerJson.JsonMessage>> GenerateMessageList(List<Message> includedMessages, Dictionary<ulong, int> userIndices, IViewerExportStrategy strategy) {
var data = new Dictionary<Snowflake, Dictionary<Snowflake, ViewerJson.JsonMessage>>();
foreach (var grouping in includedMessages.GroupBy(static message => message.Channel)) {
var channel = grouping.Key.ToString();
var channelData = new Dictionary<string, object>();
var channelIdSnowflake = new Snowflake(grouping.Key);
var channelData = new Dictionary<Snowflake, ViewerJson.JsonMessage>();
foreach (var message in grouping) {
var obj = new Dictionary<string, object> {
["u"] = userIndices[message.Sender],
["t"] = message.Timestamp,
};
if (!string.IsNullOrEmpty(message.Text)) {
obj["m"] = message.Text;
}
if (message.EditTimestamp != null) {
obj["te"] = message.EditTimestamp;
}
if (message.RepliedToId != null) {
obj["r"] = message.RepliedToId.Value;
}
if (!message.Attachments.IsEmpty) {
obj["a"] = message.Attachments.Select(attachment => {
var a = new Dictionary<string, object> {
{ "url", strategy.GetAttachmentUrl(attachment) },
{ "name", Uri.TryCreate(attachment.Url, UriKind.Absolute, out var uri) ? Path.GetFileName(uri.LocalPath) : attachment.Url },
var messageIdSnowflake = new Snowflake(message.Id);
channelData[messageIdSnowflake] = new ViewerJson.JsonMessage {
U = userIndices[message.Sender],
T = message.Timestamp,
M = string.IsNullOrEmpty(message.Text) ? null : message.Text,
Te = message.EditTimestamp,
R = message.RepliedToId?.ToString(),
A = message.Attachments.IsEmpty ? null : message.Attachments.Select(attachment => {
var a = new ViewerJson.JsonMessageAttachment {
Url = strategy.GetAttachmentUrl(attachment),
Name = Uri.TryCreate(attachment.NormalizedUrl, UriKind.Absolute, out var uri) ? Path.GetFileName(uri.LocalPath) : attachment.NormalizedUrl
};
if (attachment is { Width: not null, Height: not null }) {
a["width"] = attachment.Width;
a["height"] = attachment.Height;
a.Width = attachment.Width;
a.Height = attachment.Height;
}
return a;
}).ToArray();
}
if (!message.Embeds.IsEmpty) {
obj["e"] = message.Embeds.Select(static embed => embed.Json).ToArray();
}
if (!message.Reactions.IsEmpty) {
obj["re"] = message.Reactions.Select(static reaction => {
var r = new Dictionary<string, object>();
if (reaction.EmojiId != null) {
r["id"] = reaction.EmojiId.Value;
}
if (reaction.EmojiName != null) {
r["n"] = reaction.EmojiName;
}
r["a"] = reaction.EmojiFlags.HasFlag(EmojiFlags.Animated);
r["c"] = reaction.Count;
return r;
}).ToArray();
}
channelData[message.Id.ToString()] = obj;
}).ToArray(),
E = message.Embeds.IsEmpty ? null : message.Embeds.Select(static embed => embed.Json).ToArray(),
Re = message.Reactions.IsEmpty ? null : message.Reactions.Select(static reaction => new ViewerJson.JsonMessageReaction {
Id = reaction.EmojiId?.ToString(),
N = reaction.EmojiName,
A = reaction.EmojiFlags.HasFlag(EmojiFlags.Animated),
C = reaction.Count
}).ToArray()
};
}
data[channel] = channelData;
data[channelIdSnowflake] = channelData;
}
return data;

View File

@@ -1,15 +0,0 @@
using System;
using System.Text.Json;
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Export;
sealed class ViewerJsonSnowflakeSerializer : JsonConverter<ulong> {
public override ulong Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) {
return ulong.Parse(reader.GetString()!);
}
public override void Write(Utf8JsonWriter writer, ulong value, JsonSerializerOptions options) {
writer.WriteStringValue(value.ToString());
}
}

View File

@@ -23,7 +23,7 @@ public interface IDatabaseFile : IDisposable {
void AddMessages(Message[] messages);
int CountMessages(MessageFilter? filter = null);
List<Message> GetMessages(MessageFilter? filter = null);
List<Message> GetMessages(MessageFilter? filter = null, bool includeText = true);
HashSet<ulong> GetMessageIds(MessageFilter? filter = null);
void RemoveMessages(MessageFilter filter, FilterRemovalMode mode);

View File

@@ -0,0 +1,23 @@
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Import;
sealed class DiscordEmbedLegacyJson {
public required string Url { get; init; }
public required string Type { get; init; }
public bool DhtLegacy { get; } = true;
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Title { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Description { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public ImageJson? Image { get; init; }
public sealed class ImageJson {
public required string Url { get; init; }
}
}

View File

@@ -0,0 +1,7 @@
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Import;
[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.SnakeCaseLower, GenerationMode = JsonSourceGenerationMode.Default)]
[JsonSerializable(typeof(DiscordEmbedLegacyJson))]
sealed partial class DiscordEmbedLegacyJsonContext : JsonSerializerContext {}

View File

@@ -21,7 +21,7 @@ public static class LegacyArchiveImport {
public static async Task<bool> Read(Stream stream, IDatabaseFile db, FakeSnowflake fakeSnowflake, Func<Data.Server[], Task<Dictionary<Data.Server, ulong>?>> askForServerIds) {
var perf = Log.Start();
var root = await JsonSerializer.DeserializeAsync<JsonElement>(stream);
var root = await JsonSerializer.DeserializeAsync(stream, JsonElementContext.Default.JsonElement);
try {
var meta = root.RequireObject("meta");
@@ -197,7 +197,8 @@ public static class LegacyArchiveImport {
Id = fakeSnowflake.Next(),
Name = name,
Type = type,
Url = url,
NormalizedUrl = url,
DownloadUrl = url,
Size = 0, // unknown size
};
}).DistinctByKeyStable(static attachment => {
@@ -211,30 +212,17 @@ public static class LegacyArchiveImport {
return embedsArray.Where(static embedObj => embedObj.HasKey("url")).Select(embedObj => {
string url = embedObj.RequireString("url", path);
string type = embedObj.RequireString("type", path);
var embedJson = new Dictionary<string, object> {
{ "url", url },
{ "type", type },
{ "dht_legacy", true },
var embed = new DiscordEmbedLegacyJson {
Url = url,
Type = type,
Title = type == "rich" && embedObj.HasKey("t") ? embedObj.RequireString("t", path) : null,
Description = type == "rich" && embedObj.HasKey("d") ? embedObj.RequireString("d", path) : null,
Image = type == "image" ? new DiscordEmbedLegacyJson.ImageJson { Url = url } : null
};
if (type == "image") {
embedJson["image"] = new Dictionary<string, string> {
{ "url", url }
};
}
else if (type == "rich") {
if (embedObj.HasKey("t")) {
embedJson["title"] = embedObj.RequireString("t", path);
}
if (embedObj.HasKey("d")) {
embedJson["description"] = embedObj.RequireString("d", path);
}
}
return new Embed {
Json = JsonSerializer.Serialize(embedJson)
Json = JsonSerializer.Serialize(embed, DiscordEmbedLegacyJsonContext.Default.DiscordEmbedLegacyJson)
};
});
}

View File

@@ -0,0 +1,15 @@
using System;
using System.Threading.Tasks;
namespace DHT.Server.Database.Sqlite;
public interface ISchemaUpgradeCallbacks {
Task<bool> CanUpgrade();
Task Start(int versionSteps, Func<IProgressReporter, Task> doUpgrade);
public interface IProgressReporter {
Task NextVersion();
Task MainWork(string message, int finishedItems, int totalItems);
Task SubWork(string message, int finishedItems, int totalItems);
}
}

View File

@@ -1,13 +1,15 @@
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using DHT.Server.Database.Exceptions;
using DHT.Server.Database.Sqlite.Utils;
using DHT.Server.Download;
using DHT.Utils.Logging;
using Microsoft.Data.Sqlite;
namespace DHT.Server.Database.Sqlite;
sealed class Schema {
internal const int Version = 5;
internal const int Version = 6;
private static readonly Log Log = Log.ForType<Schema>();
@@ -17,12 +19,8 @@ sealed class Schema {
this.conn = conn;
}
private void Execute(string sql) {
conn.Command(sql).ExecuteNonQuery();
}
public async Task<bool> Setup(Func<Task<bool>> checkCanUpgradeSchemas) {
Execute(@"CREATE TABLE IF NOT EXISTS metadata (key TEXT PRIMARY KEY, value TEXT)");
public async Task<bool> Setup(ISchemaUpgradeCallbacks callbacks) {
conn.Execute(@"CREATE TABLE IF NOT EXISTS metadata (key TEXT PRIMARY KEY, value TEXT)");
var dbVersionStr = conn.SelectScalar("SELECT value FROM metadata WHERE key = 'version'");
if (dbVersionStr == null) {
@@ -35,137 +33,323 @@ sealed class Schema {
throw new DatabaseTooNewException(dbVersion);
}
else if (dbVersion < Version) {
var proceed = await checkCanUpgradeSchemas();
var proceed = await callbacks.CanUpgrade();
if (!proceed) {
return false;
}
UpgradeSchemas(dbVersion);
await callbacks.Start(Version - dbVersion, async reporter => await UpgradeSchemas(dbVersion, reporter));
}
return true;
}
private void InitializeSchemas() {
Execute(@"CREATE TABLE users (
id INTEGER PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
avatar_url TEXT,
discriminator TEXT)");
conn.Execute("""
CREATE TABLE users (
id INTEGER PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
avatar_url TEXT,
discriminator TEXT
)
""");
Execute(@"CREATE TABLE servers (
id INTEGER PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
type TEXT NOT NULL)");
conn.Execute("""
CREATE TABLE servers (
id INTEGER PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
type TEXT NOT NULL
)
""");
Execute(@"CREATE TABLE channels (
id INTEGER PRIMARY KEY NOT NULL,
server INTEGER NOT NULL,
name TEXT NOT NULL,
parent_id INTEGER,
position INTEGER,
topic TEXT,
nsfw INTEGER)");
conn.Execute("""
CREATE TABLE channels (
id INTEGER PRIMARY KEY NOT NULL,
server INTEGER NOT NULL,
name TEXT NOT NULL,
parent_id INTEGER,
position INTEGER,
topic TEXT,
nsfw INTEGER
)
""");
Execute(@"CREATE TABLE messages (
message_id INTEGER PRIMARY KEY NOT NULL,
sender_id INTEGER NOT NULL,
channel_id INTEGER NOT NULL,
text TEXT NOT NULL,
timestamp INTEGER NOT NULL)");
conn.Execute("""
CREATE TABLE messages (
message_id INTEGER PRIMARY KEY NOT NULL,
sender_id INTEGER NOT NULL,
channel_id INTEGER NOT NULL,
text TEXT NOT NULL,
timestamp INTEGER NOT NULL
)
""");
Execute(@"CREATE TABLE attachments (
message_id INTEGER NOT NULL,
attachment_id INTEGER NOT NULL PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
type TEXT,
url TEXT NOT NULL,
size INTEGER NOT NULL,
width INTEGER,
height INTEGER)");
conn.Execute("""
CREATE TABLE attachments (
message_id INTEGER NOT NULL,
attachment_id INTEGER NOT NULL PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
type TEXT,
normalized_url TEXT NOT NULL,
download_url TEXT,
size INTEGER NOT NULL,
width INTEGER,
height INTEGER
)
""");
Execute(@"CREATE TABLE embeds (
message_id INTEGER NOT NULL,
json TEXT NOT NULL)");
conn.Execute("""
CREATE TABLE embeds (
message_id INTEGER NOT NULL,
json TEXT NOT NULL
)
""");
Execute(@"CREATE TABLE reactions (
message_id INTEGER NOT NULL,
emoji_id INTEGER,
emoji_name TEXT,
emoji_flags INTEGER NOT NULL,
count INTEGER NOT NULL)");
conn.Execute("""
CREATE TABLE downloads (
normalized_url TEXT NOT NULL PRIMARY KEY,
download_url TEXT,
status INTEGER NOT NULL,
size INTEGER NOT NULL,
blob BLOB
)
""");
conn.Execute("""
CREATE TABLE reactions (
message_id INTEGER NOT NULL,
emoji_id INTEGER,
emoji_name TEXT,
emoji_flags INTEGER NOT NULL,
count INTEGER NOT NULL
)
""");
CreateMessageEditTimestampTable();
CreateMessageRepliedToTable();
CreateDownloadsTable();
Execute("CREATE INDEX attachments_message_ix ON attachments(message_id)");
Execute("CREATE INDEX embeds_message_ix ON embeds(message_id)");
Execute("CREATE INDEX reactions_message_ix ON reactions(message_id)");
conn.Execute("CREATE INDEX attachments_message_ix ON attachments(message_id)");
conn.Execute("CREATE INDEX embeds_message_ix ON embeds(message_id)");
conn.Execute("CREATE INDEX reactions_message_ix ON reactions(message_id)");
Execute("INSERT INTO metadata (key, value) VALUES ('version', " + Version + ")");
conn.Execute("INSERT INTO metadata (key, value) VALUES ('version', " + Version + ")");
}
private void CreateMessageEditTimestampTable() {
Execute(@"CREATE TABLE edit_timestamps (
message_id INTEGER PRIMARY KEY NOT NULL,
edit_timestamp INTEGER NOT NULL)");
conn.Execute("""
CREATE TABLE edit_timestamps (
message_id INTEGER PRIMARY KEY NOT NULL,
edit_timestamp INTEGER NOT NULL
)
""");
}
private void CreateMessageRepliedToTable() {
Execute(@"CREATE TABLE replied_to (
message_id INTEGER PRIMARY KEY NOT NULL,
replied_to_id INTEGER NOT NULL)");
conn.Execute("""
CREATE TABLE replied_to (
message_id INTEGER PRIMARY KEY NOT NULL,
replied_to_id INTEGER NOT NULL
)
""");
}
private void CreateDownloadsTable() {
Execute(@"CREATE TABLE downloads (
url TEXT NOT NULL PRIMARY KEY,
status INTEGER NOT NULL,
size INTEGER NOT NULL,
blob BLOB)");
private async Task NormalizeAttachmentUrls(ISchemaUpgradeCallbacks.IProgressReporter reporter) {
await reporter.SubWork("Preparing attachments...", 0, 0);
var normalizedUrls = new Dictionary<long, string>();
await using (var selectCmd = conn.Command("SELECT attachment_id, url FROM attachments")) {
await using var reader = await selectCmd.ExecuteReaderAsync();
while (reader.Read()) {
var attachmentId = reader.GetInt64(0);
var originalUrl = reader.GetString(1);
normalizedUrls[attachmentId] = DiscordCdn.NormalizeUrl(originalUrl);
}
}
await using var tx = conn.BeginTransaction();
int totalUrls = normalizedUrls.Count;
int processedUrls = -1;
await using (var updateCmd = conn.Command("UPDATE attachments SET download_url = url, url = :normalized_url WHERE attachment_id = :attachment_id")) {
updateCmd.Parameters.Add(":attachment_id", SqliteType.Integer);
updateCmd.Parameters.Add(":normalized_url", SqliteType.Text);
foreach (var (attachmentId, normalizedUrl) in normalizedUrls) {
if (++processedUrls % 1000 == 0) {
await reporter.SubWork("Updating URLs...", processedUrls, totalUrls);
}
updateCmd.Set(":attachment_id", attachmentId);
updateCmd.Set(":normalized_url", normalizedUrl);
updateCmd.ExecuteNonQuery();
}
}
await reporter.SubWork("Updating URLs...", totalUrls, totalUrls);
await tx.CommitAsync();
}
private void UpgradeSchemas(int dbVersion) {
private async Task NormalizeDownloadUrls(ISchemaUpgradeCallbacks.IProgressReporter reporter) {
await reporter.SubWork("Preparing downloads...", 0, 0);
var normalizedUrlsToOriginalUrls = new Dictionary<string, string>();
var duplicateUrlsToDelete = new HashSet<string>();
await using (var selectCmd = conn.Command("SELECT url FROM downloads ORDER BY CASE WHEN status = 200 THEN 0 ELSE 1 END")) {
await using var reader = await selectCmd.ExecuteReaderAsync();
while (reader.Read()) {
var originalUrl = reader.GetString(0);
var normalizedUrl = DiscordCdn.NormalizeUrl(originalUrl);
if (!normalizedUrlsToOriginalUrls.TryAdd(normalizedUrl, originalUrl)) {
duplicateUrlsToDelete.Add(originalUrl);
}
}
}
conn.Execute("PRAGMA cache_size = -20000");
SqliteTransaction tx;
await using (tx = conn.BeginTransaction()) {
await reporter.SubWork("Deleting duplicates...", 0, 0);
await using (var deleteCmd = conn.Delete("downloads", ("url", SqliteType.Text))) {
foreach (var duplicateUrl in duplicateUrlsToDelete) {
deleteCmd.Set(":url", duplicateUrl);
deleteCmd.ExecuteNonQuery();
}
}
await tx.CommitAsync();
}
int totalUrls = normalizedUrlsToOriginalUrls.Count;
int processedUrls = -1;
tx = conn.BeginTransaction();
await using (var updateCmd = conn.Command("UPDATE downloads SET download_url = :download_url, url = :normalized_url WHERE url = :download_url")) {
updateCmd.Parameters.Add(":normalized_url", SqliteType.Text);
updateCmd.Parameters.Add(":download_url", SqliteType.Text);
foreach (var (normalizedUrl, downloadUrl) in normalizedUrlsToOriginalUrls) {
if (++processedUrls % 100 == 0) {
await reporter.SubWork("Updating URLs...", processedUrls, totalUrls);
// Not proper way of dealing with transactions, but it avoids a long commit at the end.
// Schema upgrades are already non-atomic anyways, so this doesn't make it worse.
await tx.CommitAsync();
await tx.DisposeAsync();
tx = conn.BeginTransaction();
updateCmd.Transaction = tx;
}
updateCmd.Set(":normalized_url", normalizedUrl);
updateCmd.Set(":download_url", downloadUrl);
updateCmd.ExecuteNonQuery();
}
}
await reporter.SubWork("Updating URLs...", totalUrls, totalUrls);
await tx.CommitAsync();
await tx.DisposeAsync();
conn.Execute("PRAGMA cache_size = -2000");
}
private async Task UpgradeSchemas(int dbVersion, ISchemaUpgradeCallbacks.IProgressReporter reporter) {
var perf = Log.Start("from version " + dbVersion);
Execute("UPDATE metadata SET value = " + Version + " WHERE key = 'version'");
conn.Execute("UPDATE metadata SET value = " + Version + " WHERE key = 'version'");
if (dbVersion <= 1) {
Execute("ALTER TABLE channels ADD parent_id INTEGER");
await reporter.MainWork("Applying schema changes...", 0, 1);
conn.Execute("ALTER TABLE channels ADD parent_id INTEGER");
perf.Step("Upgrade to version 2");
await reporter.NextVersion();
}
if (dbVersion <= 2) {
await reporter.MainWork("Applying schema changes...", 0, 1);
CreateMessageEditTimestampTable();
CreateMessageRepliedToTable();
Execute(@"INSERT INTO edit_timestamps (message_id, edit_timestamp)
SELECT message_id, edit_timestamp FROM messages
WHERE edit_timestamp IS NOT NULL");
conn.Execute("""
INSERT INTO edit_timestamps (message_id, edit_timestamp)
SELECT message_id, edit_timestamp
FROM messages
WHERE edit_timestamp IS NOT NULL
""");
Execute(@"INSERT INTO replied_to (message_id, replied_to_id)
SELECT message_id, replied_to_id FROM messages
WHERE replied_to_id IS NOT NULL");
conn.Execute("""
INSERT INTO replied_to (message_id, replied_to_id)
SELECT message_id, replied_to_id
FROM messages
WHERE replied_to_id IS NOT NULL
""");
Execute("ALTER TABLE messages DROP COLUMN replied_to_id");
Execute("ALTER TABLE messages DROP COLUMN edit_timestamp");
conn.Execute("ALTER TABLE messages DROP COLUMN replied_to_id");
conn.Execute("ALTER TABLE messages DROP COLUMN edit_timestamp");
perf.Step("Upgrade to version 3");
Execute("VACUUM");
await reporter.MainWork("Vacuuming the database...", 1, 1);
conn.Execute("VACUUM");
perf.Step("Vacuum");
await reporter.NextVersion();
}
if (dbVersion <= 3) {
CreateDownloadsTable();
conn.Execute("""
CREATE TABLE downloads (
url TEXT NOT NULL PRIMARY KEY,
status INTEGER NOT NULL,
size INTEGER NOT NULL,
blob BLOB
)
""");
perf.Step("Upgrade to version 4");
await reporter.NextVersion();
}
if (dbVersion <= 4) {
Execute("ALTER TABLE attachments ADD width INTEGER");
Execute("ALTER TABLE attachments ADD height INTEGER");
await reporter.MainWork("Applying schema changes...", 0, 1);
conn.Execute("ALTER TABLE attachments ADD width INTEGER");
conn.Execute("ALTER TABLE attachments ADD height INTEGER");
perf.Step("Upgrade to version 5");
await reporter.NextVersion();
}
if (dbVersion <= 5) {
await reporter.MainWork("Applying schema changes...", 0, 3);
conn.Execute("ALTER TABLE attachments ADD download_url TEXT");
conn.Execute("ALTER TABLE downloads ADD download_url TEXT");
await reporter.MainWork("Updating attachments...", 1, 3);
await NormalizeAttachmentUrls(reporter);
await reporter.MainWork("Updating downloads...", 2, 3);
await NormalizeDownloadUrls(reporter);
await reporter.MainWork("Applying schema changes...", 3, 3);
conn.Execute("ALTER TABLE attachments RENAME COLUMN url TO normalized_url");
conn.Execute("ALTER TABLE downloads RENAME COLUMN url TO normalized_url");
perf.Step("Upgrade to version 6");
await reporter.NextVersion();
}
perf.End();

View File

@@ -18,7 +18,7 @@ namespace DHT.Server.Database.Sqlite;
public sealed class SqliteDatabaseFile : IDatabaseFile {
private const int DefaultPoolSize = 5;
public static async Task<SqliteDatabaseFile?> OpenOrCreate(string path, Func<Task<bool>> checkCanUpgradeSchemas) {
public static async Task<SqliteDatabaseFile?> OpenOrCreate(string path, ISchemaUpgradeCallbacks schemaUpgradeCallbacks, TaskScheduler computeTaskResultScheduler) {
var connectionString = new SqliteConnectionStringBuilder {
DataSource = path,
Mode = SqliteOpenMode.ReadWriteCreate,
@@ -27,12 +27,16 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
var pool = new SqliteConnectionPool(connectionString, DefaultPoolSize);
bool wasOpened;
using (var conn = pool.Take()) {
wasOpened = await new Schema(conn).Setup(checkCanUpgradeSchemas);
try {
using var conn = pool.Take();
wasOpened = await new Schema(conn).Setup(schemaUpgradeCallbacks);
} catch (Exception) {
pool.Dispose();
throw;
}
if (wasOpened) {
return new SqliteDatabaseFile(path, pool);
return new SqliteDatabaseFile(path, pool, computeTaskResultScheduler);
}
else {
pool.Dispose();
@@ -49,13 +53,13 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
private readonly AsyncValueComputer<long>.Single totalAttachmentsComputer;
private readonly AsyncValueComputer<long>.Single totalDownloadsComputer;
private SqliteDatabaseFile(string path, SqliteConnectionPool pool) {
private SqliteDatabaseFile(string path, SqliteConnectionPool pool, TaskScheduler computeTaskResultScheduler) {
this.log = Log.ForType(typeof(SqliteDatabaseFile), System.IO.Path.GetFileName(path));
this.pool = pool;
this.totalMessagesComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateMessageStatistics).WithOutdatedResults().BuildWithComputer(ComputeMessageStatistics);
this.totalAttachmentsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateAttachmentStatistics).WithOutdatedResults().BuildWithComputer(ComputeAttachmentStatistics);
this.totalDownloadsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateDownloadStatistics).WithOutdatedResults().BuildWithComputer(ComputeDownloadStatistics);
this.totalMessagesComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateMessageStatistics, computeTaskResultScheduler).WithOutdatedResults().BuildWithComputer(ComputeMessageStatistics);
this.totalAttachmentsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateAttachmentStatistics, computeTaskResultScheduler).WithOutdatedResults().BuildWithComputer(ComputeAttachmentStatistics);
this.totalDownloadsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateDownloadStatistics, computeTaskResultScheduler).WithOutdatedResults().BuildWithComputer(ComputeDownloadStatistics);
this.Path = path;
this.Statistics = new DatabaseStatistics();
@@ -252,7 +256,8 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
("attachment_id", SqliteType.Integer),
("name", SqliteType.Text),
("type", SqliteType.Text),
("url", SqliteType.Text),
("normalized_url", SqliteType.Text),
("download_url", SqliteType.Text),
("size", SqliteType.Integer),
("width", SqliteType.Integer),
("height", SqliteType.Integer),
@@ -308,7 +313,8 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
attachmentCmd.Set(":attachment_id", attachment.Id);
attachmentCmd.Set(":name", attachment.Name);
attachmentCmd.Set(":type", attachment.Type);
attachmentCmd.Set(":url", attachment.Url);
attachmentCmd.Set(":normalized_url", attachment.NormalizedUrl);
attachmentCmd.Set(":download_url", attachment.DownloadUrl);
attachmentCmd.Set(":size", attachment.Size);
attachmentCmd.Set(":width", attachment.Width);
attachmentCmd.Set(":height", attachment.Height);
@@ -354,7 +360,7 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
return reader.Read() ? reader.GetInt32(0) : 0;
}
public List<Message> GetMessages(MessageFilter? filter = null) {
public List<Message> GetMessages(MessageFilter? filter = null, bool includeText = true) {
var perf = log.Start();
var list = new List<Message>();
@@ -363,11 +369,13 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
var reactions = GetAllReactions();
using var conn = pool.Take();
using var cmd = conn.Command(@"
SELECT m.message_id, m.sender_id, m.channel_id, m.text, m.timestamp, et.edit_timestamp, rt.replied_to_id
FROM messages m
LEFT JOIN edit_timestamps et ON m.message_id = et.message_id
LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereClause("m"));
using var cmd = conn.Command($"""
SELECT m.message_id, m.sender_id, m.channel_id, {(includeText ? "m.text" : "NULL")}, m.timestamp, et.edit_timestamp, rt.replied_to_id
FROM messages m
LEFT JOIN edit_timestamps et ON m.message_id = et.message_id
LEFT JOIN replied_to rt ON m.message_id = rt.message_id
{filter.GenerateWhereClause("m")}
""");
using var reader = cmd.ExecuteReader();
while (reader.Read()) {
@@ -377,7 +385,7 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
Id = id,
Sender = reader.GetUint64(1),
Channel = reader.GetUint64(2),
Text = reader.GetString(3),
Text = includeText ? reader.GetString(3) : string.Empty,
Timestamp = reader.GetInt64(4),
EditTimestamp = reader.IsDBNull(5) ? null : reader.GetInt64(5),
RepliedToId = reader.IsDBNull(6) ? null : reader.GetUint64(6),
@@ -418,7 +426,7 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
public int CountAttachments(AttachmentFilter? filter = null) {
using var conn = pool.Take();
using var cmd = conn.Command("SELECT COUNT(DISTINCT url) FROM attachments a" + filter.GenerateWhereClause("a"));
using var cmd = conn.Command("SELECT COUNT(DISTINCT normalized_url) FROM attachments a" + filter.GenerateWhereClause("a"));
using var reader = cmd.ExecuteReader();
return reader.Read() ? reader.GetInt32(0) : 0;
@@ -427,13 +435,15 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
public void AddDownload(Data.Download download) {
using var conn = pool.Take();
using var cmd = conn.Upsert("downloads", new[] {
("url", SqliteType.Text),
("normalized_url", SqliteType.Text),
("download_url", SqliteType.Text),
("status", SqliteType.Integer),
("size", SqliteType.Integer),
("blob", SqliteType.Blob),
});
cmd.Set(":url", download.Url);
cmd.Set(":normalized_url", download.NormalizedUrl);
cmd.Set(":download_url", download.DownloadUrl);
cmd.Set(":status", (int) download.Status);
cmd.Set(":size", download.Size);
cmd.Set(":blob", download.Data);
@@ -446,15 +456,16 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
var list = new List<Data.Download>();
using var conn = pool.Take();
using var cmd = conn.Command("SELECT url, status, size FROM downloads");
using var cmd = conn.Command("SELECT normalized_url, download_url, status, size FROM downloads");
using var reader = cmd.ExecuteReader();
while (reader.Read()) {
string url = reader.GetString(0);
var status = (DownloadStatus) reader.GetInt32(1);
ulong size = reader.GetUint64(2);
string normalizedUrl = reader.GetString(0);
string downloadUrl = reader.GetString(1);
var status = (DownloadStatus) reader.GetInt32(2);
ulong size = reader.GetUint64(3);
list.Add(new Data.Download(url, status, size));
list.Add(new Data.Download(normalizedUrl, downloadUrl, status, size));
}
return list;
@@ -462,8 +473,8 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
public Data.Download GetDownloadWithData(Data.Download download) {
using var conn = pool.Take();
using var cmd = conn.Command("SELECT blob FROM downloads WHERE url = :url");
cmd.AddAndSet(":url", SqliteType.Text, download.Url);
using var cmd = conn.Command("SELECT blob FROM downloads WHERE normalized_url = :url");
cmd.AddAndSet(":url", SqliteType.Text, download.NormalizedUrl);
using var reader = cmd.ExecuteReader();
@@ -475,14 +486,15 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
}
}
public DownloadedAttachment? GetDownloadedAttachment(string url) {
public DownloadedAttachment? GetDownloadedAttachment(string normalizedUrl) {
using var conn = pool.Take();
using var cmd = conn.Command(@"
SELECT a.type, d.blob FROM downloads d
LEFT JOIN attachments a ON d.url = a.url
WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
using var cmd = conn.Command("""
SELECT a.type, d.blob FROM downloads d
LEFT JOIN attachments a ON d.normalized_url = a.normalized_url
WHERE d.normalized_url = :normalized_url AND d.status = :success AND d.blob IS NOT NULL
""");
cmd.AddAndSet(":url", SqliteType.Text, url);
cmd.AddAndSet(":normalized_url", SqliteType.Text, normalizedUrl);
cmd.AddAndSet(":success", SqliteType.Integer, (int) DownloadStatus.Success);
using var reader = cmd.ExecuteReader();
@@ -499,7 +511,13 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
public void EnqueueDownloadItems(AttachmentFilter? filter = null) {
using var conn = pool.Take();
using var cmd = conn.Command("INSERT INTO downloads (url, status, size) SELECT a.url, :enqueued, MAX(a.size) FROM attachments a" + filter.GenerateWhereClause("a") + " GROUP BY a.url");
using var cmd = conn.Command($"""
INSERT INTO downloads (normalized_url, download_url, status, size)
SELECT a.normalized_url, a.download_url, :enqueued, MAX(a.size)
FROM attachments a
{filter.GenerateWhereClause("a")}
GROUP BY a.normalized_url
""");
cmd.AddAndSet(":enqueued", SqliteType.Integer, (int) DownloadStatus.Enqueued);
cmd.ExecuteNonQuery();
}
@@ -508,7 +526,7 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
var list = new List<DownloadItem>();
using var conn = pool.Take();
using var cmd = conn.Command("SELECT url, size FROM downloads WHERE status = :enqueued LIMIT :limit");
using var cmd = conn.Command("SELECT normalized_url, download_url, size FROM downloads WHERE status = :enqueued LIMIT :limit");
cmd.AddAndSet(":enqueued", SqliteType.Integer, (int) DownloadStatus.Enqueued);
cmd.AddAndSet(":limit", SqliteType.Integer, Math.Max(0, count));
@@ -516,8 +534,9 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
while (reader.Read()) {
list.Add(new DownloadItem {
Url = reader.GetString(0),
Size = reader.GetUint64(1),
NormalizedUrl = reader.GetString(0),
DownloadUrl = reader.GetString(1),
Size = reader.GetUint64(2),
});
}
@@ -531,7 +550,7 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
public DownloadStatusStatistics GetDownloadStatusStatistics() {
static void LoadUndownloadedStatistics(ISqliteConnection conn, DownloadStatusStatistics result) {
using var cmd = conn.Command("SELECT IFNULL(COUNT(size), 0), IFNULL(SUM(size), 0) FROM (SELECT MAX(a.size) size FROM attachments a WHERE a.url NOT IN (SELECT d.url FROM downloads d) GROUP BY a.url)");
using var cmd = conn.Command("SELECT IFNULL(COUNT(size), 0), IFNULL(SUM(size), 0) FROM (SELECT MAX(a.size) size FROM attachments a WHERE a.normalized_url NOT IN (SELECT d.normalized_url FROM downloads d) GROUP BY a.normalized_url)");
using var reader = cmd.ExecuteReader();
if (reader.Read()) {
@@ -541,14 +560,16 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
}
static void LoadSuccessStatistics(ISqliteConnection conn, DownloadStatusStatistics result) {
using var cmd = conn.Command(@"SELECT
IFNULL(SUM(CASE WHEN status = :enqueued THEN 1 ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status = :enqueued THEN size ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status = :success THEN 1 ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status = :success THEN size ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status != :enqueued AND status != :success THEN 1 ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status != :enqueued AND status != :success THEN size ELSE 0 END), 0)
FROM downloads");
using var cmd = conn.Command("""
SELECT
IFNULL(SUM(CASE WHEN status = :enqueued THEN 1 ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status = :enqueued THEN size ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status = :success THEN 1 ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status = :success THEN size ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status != :enqueued AND status != :success THEN 1 ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status != :enqueued AND status != :success THEN size ELSE 0 END), 0)
FROM downloads
""");
cmd.AddAndSet(":enqueued", SqliteType.Integer, (int) DownloadStatus.Enqueued);
cmd.AddAndSet(":success", SqliteType.Integer, (int) DownloadStatus.Success);
@@ -576,7 +597,7 @@ FROM downloads");
var dict = new MultiDictionary<ulong, Attachment>();
using var conn = pool.Take();
using var cmd = conn.Command("SELECT message_id, attachment_id, name, type, url, size, width, height FROM attachments");
using var cmd = conn.Command("SELECT message_id, attachment_id, name, type, normalized_url, download_url, size, width, height FROM attachments");
using var reader = cmd.ExecuteReader();
while (reader.Read()) {
@@ -586,10 +607,11 @@ FROM downloads");
Id = reader.GetUint64(1),
Name = reader.GetString(2),
Type = reader.IsDBNull(3) ? null : reader.GetString(3),
Url = reader.GetString(4),
Size = reader.GetUint64(5),
Width = reader.IsDBNull(6) ? null : reader.GetInt32(6),
Height = reader.IsDBNull(7) ? null : reader.GetInt32(7),
NormalizedUrl = reader.GetString(4),
DownloadUrl = reader.GetString(5),
Size = reader.GetUint64(6),
Width = reader.IsDBNull(7) ? null : reader.GetInt32(7),
Height = reader.IsDBNull(8) ? null : reader.GetInt32(8),
});
}
@@ -677,7 +699,7 @@ FROM downloads");
private long ComputeAttachmentStatistics() {
using var conn = pool.Take();
return conn.SelectScalar("SELECT COUNT(DISTINCT url) FROM attachments") as long? ?? 0L;
return conn.SelectScalar("SELECT COUNT(DISTINCT normalized_url) FROM attachments") as long? ?? 0L;
}
private void UpdateAttachmentStatistics(long totalAttachments) {

View File

@@ -50,10 +50,10 @@ static class SqliteFilters {
}
if (filter.DownloadItemRule == AttachmentFilter.DownloadItemRules.OnlyNotPresent) {
where.AddCondition("url NOT IN (SELECT url FROM downloads)");
where.AddCondition("normalized_url NOT IN (SELECT normalized_url FROM downloads)");
}
else if (filter.DownloadItemRule == AttachmentFilter.DownloadItemRules.OnlyPresent) {
where.AddCondition("url IN (SELECT url FROM downloads)");
where.AddCondition("normalized_url IN (SELECT normalized_url FROM downloads)");
}
return where.Generate();

View File

@@ -5,14 +5,19 @@ using Microsoft.Data.Sqlite;
namespace DHT.Server.Database.Sqlite.Utils;
static class SqliteExtensions {
public static SqliteTransaction BeginTransaction(this ISqliteConnection conn) {
return conn.InnerConnection.BeginTransaction();
}
public static SqliteCommand Command(this ISqliteConnection conn, string sql) {
var cmd = conn.InnerConnection.CreateCommand();
cmd.CommandText = sql;
return cmd;
}
public static SqliteTransaction BeginTransaction(this ISqliteConnection conn) {
return conn.InnerConnection.BeginTransaction();
public static void Execute(this ISqliteConnection conn, string sql) {
using var cmd = conn.Command(sql);
cmd.ExecuteNonQuery();
}
public static object? SelectScalar(this ISqliteConnection conn, string sql) {

View File

@@ -87,16 +87,16 @@ public sealed class BackgroundDownloadThread : BaseModel {
FillQueue(db, queue, cancellationToken);
while (!cancellationToken.IsCancellationRequested && queue.TryDequeue(out var item)) {
var url = item.Url;
Log.Debug("Downloading " + url + "...");
var downloadUrl = item.DownloadUrl;
Log.Debug("Downloading " + downloadUrl + "...");
try {
db.AddDownload(Data.Download.NewSuccess(url, await client.GetByteArrayAsync(url, cancellationToken)));
db.AddDownload(Data.Download.NewSuccess(item, await client.GetByteArrayAsync(downloadUrl, cancellationToken)));
} catch (HttpRequestException e) {
db.AddDownload(Data.Download.NewFailure(url, e.StatusCode, item.Size));
db.AddDownload(Data.Download.NewFailure(item, e.StatusCode, item.Size));
Log.Error(e);
} catch (Exception e) {
db.AddDownload(Data.Download.NewFailure(url, null, item.Size));
db.AddDownload(Data.Download.NewFailure(item, null, item.Size));
Log.Error(e);
} finally {
parameters.FireOnItemFinished(item);

View File

@@ -0,0 +1,15 @@
using System;
using System.Collections.Frozen;
namespace DHT.Server.Download;
static class DiscordCdn {
private static FrozenSet<string> CdnHosts { get; } = new [] {
"cdn.discordapp.com",
"cdn.discord.com",
}.ToFrozenSet();
public static string NormalizeUrl(string originalUrl) {
return Uri.TryCreate(originalUrl, UriKind.Absolute, out var uri) && CdnHosts.Contains(uri.Host) ? uri.GetLeftPart(UriPartial.Path) : originalUrl;
}
}

View File

@@ -1,6 +1,7 @@
namespace DHT.Server.Download;
public readonly struct DownloadItem {
public string Url { get; init; }
public string NormalizedUrl { get; init; }
public string DownloadUrl { get; init; }
public ulong Size { get; init; }
}

View File

@@ -3,12 +3,9 @@ using System.Net;
using System.Text.Json;
using System.Threading.Tasks;
using DHT.Server.Database;
using DHT.Server.Service;
using DHT.Utils.Http;
using DHT.Utils.Logging;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Extensions;
using Microsoft.Extensions.Primitives;
namespace DHT.Server.Endpoints;
@@ -16,25 +13,14 @@ abstract class BaseEndpoint {
private static readonly Log Log = Log.ForType<BaseEndpoint>();
protected IDatabaseFile Db { get; }
private readonly ServerParameters parameters;
protected BaseEndpoint(IDatabaseFile db, ServerParameters parameters) {
protected BaseEndpoint(IDatabaseFile db) {
this.Db = db;
this.parameters = parameters;
}
private async Task Handle(HttpContext ctx, StringValues token) {
var request = ctx.Request;
public async Task Handle(HttpContext ctx) {
var response = ctx.Response;
Log.Info("Request: " + request.GetDisplayUrl() + " (" + request.ContentLength + " B)");
if (token.Count != 1 || token[0] != parameters.Token) {
Log.Error("Token: " + (token.Count == 1 ? token[0] : "<missing>"));
response.StatusCode = (int) HttpStatusCode.Forbidden;
return;
}
try {
response.StatusCode = (int) HttpStatusCode.OK;
var output = await Respond(ctx);
@@ -49,17 +35,13 @@ abstract class BaseEndpoint {
}
}
public async Task HandleGet(HttpContext ctx) {
await Handle(ctx, ctx.Request.Query["token"]);
}
public async Task HandlePost(HttpContext ctx) {
await Handle(ctx, ctx.Request.Headers["X-DHT-Token"]);
}
protected abstract Task<IHttpOutput> Respond(HttpContext ctx);
protected static async Task<JsonElement> ReadJson(HttpContext ctx) {
return await ctx.Request.ReadFromJsonAsync<JsonElement?>() ?? throw new HttpException(HttpStatusCode.UnsupportedMediaType, "This endpoint only accepts JSON.");
try {
return await ctx.Request.ReadFromJsonAsync(JsonElementContext.Default.JsonElement);
} catch (JsonException) {
throw new HttpException(HttpStatusCode.UnsupportedMediaType, "This endpoint only accepts JSON.");
}
}
}

View File

@@ -2,14 +2,13 @@ using System.Net;
using System.Threading.Tasks;
using DHT.Server.Data;
using DHT.Server.Database;
using DHT.Server.Service;
using DHT.Utils.Http;
using Microsoft.AspNetCore.Http;
namespace DHT.Server.Endpoints;
sealed class GetAttachmentEndpoint : BaseEndpoint {
public GetAttachmentEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db, parameters) {}
public GetAttachmentEndpoint(IDatabaseFile db) : base(db) {}
protected override Task<IHttpOutput> Respond(HttpContext ctx) {
string attachmentUrl = WebUtility.UrlDecode((string) ctx.Request.RouteValues["url"]!);

View File

@@ -0,0 +1,34 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Threading.Tasks;
using DHT.Server.Data.Filters;
using DHT.Server.Database;
using DHT.Utils.Http;
using Microsoft.AspNetCore.Http;
using GetMessagesJsonContext = DHT.Server.Endpoints.Responses.GetMessagesJsonContext;
namespace DHT.Server.Endpoints;
sealed class GetMessagesEndpoint : BaseEndpoint {
public GetMessagesEndpoint(IDatabaseFile db) : base(db) {}
protected override Task<IHttpOutput> Respond(HttpContext ctx) {
HashSet<ulong> messageIdSet;
try {
var messageIds = ctx.Request.Query["id"];
messageIdSet = messageIds.Select(ulong.Parse!).ToHashSet();
} catch (Exception) {
throw new HttpException(HttpStatusCode.BadRequest, "Invalid message ids.");
}
var messageFilter = new MessageFilter {
MessageIds = messageIdSet
};
var messages = Db.GetMessages(messageFilter).ToDictionary(static message => message.Id, static message => message.Text);
var response = new HttpOutput.Json<Dictionary<ulong, string>>(messages, GetMessagesJsonContext.Default.DictionaryUInt64String);
return Task.FromResult<IHttpOutput>(response);
}
}

View File

@@ -0,0 +1,34 @@
using System.Reflection;
using System.Text;
using System.Threading.Tasks;
using System.Web;
using DHT.Server.Database;
using DHT.Server.Service;
using DHT.Utils.Http;
using DHT.Utils.Resources;
using Microsoft.AspNetCore.Http;
namespace DHT.Server.Endpoints;
sealed class GetTrackingScriptEndpoint : BaseEndpoint {
private static ResourceLoader Resources { get; } = new (Assembly.GetExecutingAssembly());
private readonly ServerParameters serverParameters;
public GetTrackingScriptEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db) {
serverParameters = parameters;
}
protected override async Task<IHttpOutput> Respond(HttpContext ctx) {
string bootstrap = await Resources.ReadTextAsync("Tracker/bootstrap.js");
string script = bootstrap.Replace("= 0; /*[PORT]*/", "= " + serverParameters.Port + ";")
.Replace("/*[TOKEN]*/", HttpUtility.JavaScriptStringEncode(serverParameters.Token))
.Replace("/*[IMPORTS]*/", await Resources.ReadJoinedAsync("Tracker/scripts/", '\n'))
.Replace("/*[CSS-CONTROLLER]*/", await Resources.ReadTextAsync("Tracker/styles/controller.css"))
.Replace("/*[CSS-SETTINGS]*/", await Resources.ReadTextAsync("Tracker/styles/settings.css"))
.Replace("/*[DEBUGGER]*/", ctx.Request.Query.ContainsKey("debug") ? "debugger;" : "");
ctx.Response.Headers.Append("X-DHT", "1");
return new HttpOutput.File("text/javascript", Encoding.UTF8.GetBytes(script));
}
}

View File

@@ -0,0 +1,8 @@
using System.Collections.Generic;
using System.Text.Json.Serialization;
namespace DHT.Server.Endpoints.Responses;
[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase, GenerationMode = JsonSourceGenerationMode.Default)]
[JsonSerializable(typeof(Dictionary<ulong, string>))]
sealed partial class GetMessagesJsonContext : JsonSerializerContext {}

View File

@@ -3,14 +3,13 @@ using System.Text.Json;
using System.Threading.Tasks;
using DHT.Server.Data;
using DHT.Server.Database;
using DHT.Server.Service;
using DHT.Utils.Http;
using Microsoft.AspNetCore.Http;
namespace DHT.Server.Endpoints;
sealed class TrackChannelEndpoint : BaseEndpoint {
public TrackChannelEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db, parameters) {}
public TrackChannelEndpoint(IDatabaseFile db) : base(db) {}
protected override async Task<IHttpOutput> Respond(HttpContext ctx) {
var root = await ReadJson(ctx);

View File

@@ -8,7 +8,7 @@ using System.Threading.Tasks;
using DHT.Server.Data;
using DHT.Server.Data.Filters;
using DHT.Server.Database;
using DHT.Server.Service;
using DHT.Server.Download;
using DHT.Utils.Collections;
using DHT.Utils.Http;
using Microsoft.AspNetCore.Http;
@@ -16,7 +16,10 @@ using Microsoft.AspNetCore.Http;
namespace DHT.Server.Endpoints;
sealed class TrackMessagesEndpoint : BaseEndpoint {
public TrackMessagesEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db, parameters) {}
private const string HasNewMessages = "1";
private const string NoNewMessages = "0";
public TrackMessagesEndpoint(IDatabaseFile db) : base(db) {}
protected override async Task<IHttpOutput> Respond(HttpContext ctx) {
var root = await ReadJson(ctx);
@@ -40,7 +43,7 @@ sealed class TrackMessagesEndpoint : BaseEndpoint {
Db.AddMessages(messages);
return new HttpOutput.Json(anyNewMessages ? 1 : 0);
return new HttpOutput.Text(anyNewMessages ? HasNewMessages : NoNewMessages);
}
private static Message ReadMessage(JsonElement json, string path) => new() {
@@ -57,14 +60,18 @@ sealed class TrackMessagesEndpoint : BaseEndpoint {
};
[SuppressMessage("ReSharper", "ConvertToLambdaExpression")]
private static IEnumerable<Attachment> ReadAttachments(JsonElement.ArrayEnumerator array, string path) => array.Select(ele => new Attachment {
Id = ele.RequireSnowflake("id", path),
Name = ele.RequireString("name", path),
Type = ele.HasKey("type") ? ele.RequireString("type", path) : null,
Url = ele.RequireString("url", path),
Size = (ulong) ele.RequireLong("size", path),
Width = ele.HasKey("width") ? ele.RequireInt("width", path) : null,
Height = ele.HasKey("height") ? ele.RequireInt("height", path) : null,
private static IEnumerable<Attachment> ReadAttachments(JsonElement.ArrayEnumerator array, string path) => array.Select(ele => {
var downloadUrl = ele.RequireString("url", path);
return new Attachment {
Id = ele.RequireSnowflake("id", path),
Name = ele.RequireString("name", path),
Type = ele.HasKey("type") ? ele.RequireString("type", path) : null,
NormalizedUrl = DiscordCdn.NormalizeUrl(downloadUrl),
DownloadUrl = downloadUrl,
Size = (ulong) ele.RequireLong("size", path),
Width = ele.HasKey("width") ? ele.RequireInt("width", path) : null,
Height = ele.HasKey("height") ? ele.RequireInt("height", path) : null,
};
}).DistinctByKeyStable(static attachment => {
// Some Discord messages have duplicate attachments with the same id for unknown reasons.
return attachment.Id;

View File

@@ -3,14 +3,13 @@ using System.Text.Json;
using System.Threading.Tasks;
using DHT.Server.Data;
using DHT.Server.Database;
using DHT.Server.Service;
using DHT.Utils.Http;
using Microsoft.AspNetCore.Http;
namespace DHT.Server.Endpoints;
sealed class TrackUsersEndpoint : BaseEndpoint {
public TrackUsersEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db, parameters) {}
public TrackUsersEndpoint(IDatabaseFile db) : base(db) {}
protected override async Task<IHttpOutput> Respond(HttpContext ctx) {
var root = await ReadJson(ctx);

View File

@@ -1,30 +1,43 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<RootNamespace>DHT.Server</RootNamespace>
<Nullable>enable</Nullable>
<AssemblyName>DiscordHistoryTracker.Server</AssemblyName>
<PackageId>DiscordHistoryTrackerServer</PackageId>
<Authors>chylex</Authors>
<Company>DiscordHistoryTracker</Company>
<Product>DiscordHistoryTrackerServer</Product>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<DebugSymbols>true</DebugSymbols>
<DebugType>none</DebugType>
</PropertyGroup>
<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.Sqlite" Version="7.0.9" />
<PackageReference Include="Microsoft.Data.Sqlite" Version="8.0.0" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\Utils\Utils.csproj" />
</ItemGroup>
<ItemGroup>
<Compile Include="..\Version.cs" Link="Version.cs" />
</ItemGroup>
<ItemGroup>
<EmbeddedResource Include="../Resources/Tracker/bootstrap.js">
<LogicalName>Tracker\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
</EmbeddedResource>
<EmbeddedResource Include="../Resources/Tracker/scripts/**">
<LogicalName>Tracker\scripts\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/scripts/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
</EmbeddedResource>
<EmbeddedResource Include="../Resources/Tracker/styles/**">
<LogicalName>Tracker\styles\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/styles/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
</EmbeddedResource>
</ItemGroup>
</Project>

View File

@@ -0,0 +1,44 @@
using System.Net;
using System.Threading.Tasks;
using DHT.Utils.Logging;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Primitives;
namespace DHT.Server.Service.Middlewares;
sealed class ServerAuthorizationMiddleware {
private static readonly Log Log = Log.ForType<ServerAuthorizationMiddleware>();
private readonly RequestDelegate next;
private readonly ServerParameters serverParameters;
public ServerAuthorizationMiddleware(RequestDelegate next, ServerParameters serverParameters) {
this.next = next;
this.serverParameters = serverParameters;
}
public async Task InvokeAsync(HttpContext context) {
var request = context.Request;
bool success = HttpMethods.IsGet(request.Method)
? CheckToken(request.Query["token"])
: CheckToken(request.Headers["X-DHT-Token"]);
if (success) {
await next(context);
}
else {
context.Response.StatusCode = (int) HttpStatusCode.Forbidden;
}
}
private bool CheckToken(StringValues token) {
if (token.Count == 1 && token[0] == serverParameters.Token) {
return true;
}
else {
Log.Error("Invalid token: " + (token.Count == 1 ? token[0] : "<missing>"));
return false;
}
}
}

View File

@@ -0,0 +1,29 @@
using System.Diagnostics;
using System.Threading.Tasks;
using DHT.Utils.Logging;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Extensions;
namespace DHT.Server.Service.Middlewares;
sealed class ServerLoggingMiddleware {
private static readonly Log Log = Log.ForType<ServerLoggingMiddleware>();
private readonly RequestDelegate next;
public ServerLoggingMiddleware(RequestDelegate next) {
this.next = next;
}
public async Task InvokeAsync(HttpContext context) {
var stopwatch = Stopwatch.StartNew();
await next(context);
stopwatch.Stop();
var request = context.Request;
var requestLength = request.ContentLength ?? 0L;
var responseStatus = context.Response.StatusCode;
var elapsedMs = stopwatch.ElapsedMilliseconds;
Log.Debug("Request to " + request.GetEncodedPathAndQuery() + " (" + requestLength + " B) returned " + responseStatus + ", took " + elapsedMs + " ms");
}
}

View File

@@ -4,7 +4,6 @@ using System.Diagnostics.CodeAnalysis;
using System.Threading;
using DHT.Server.Database;
using DHT.Utils.Logging;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.DependencyInjection;
@@ -61,14 +60,12 @@ public static class ServerLauncher {
}
}
private static void StartServerFromManagementThread(int port, string token, IDatabaseFile db) {
private static void StartServerFromManagementThread(ushort port, string token, IDatabaseFile db) {
Log.Info("Starting server on port " + port + "...");
void AddServices(IServiceCollection services) {
services.AddSingleton(typeof(IDatabaseFile), db);
services.AddSingleton(typeof(ServerParameters), new ServerParameters {
Token = token
});
services.AddSingleton(typeof(ServerParameters), new ServerParameters(port, token));
}
void SetKestrelOptions(KestrelServerOptions options) {
@@ -77,11 +74,11 @@ public static class ServerLauncher {
options.ListenLocalhost(port, static listenOptions => listenOptions.Protocols = HttpProtocols.Http1);
}
Server = WebHost.CreateDefaultBuilder()
.ConfigureServices(AddServices)
.UseKestrel(SetKestrelOptions)
.UseStartup<Startup>()
.Build();
Server = new WebHostBuilder()
.ConfigureServices(AddServices)
.UseKestrel(SetKestrelOptions)
.UseStartup<Startup>()
.Build();
Server.Start();
@@ -103,7 +100,7 @@ public static class ServerLauncher {
}
}
public static void Relaunch(int port, string token, IDatabaseFile db) {
public static void Relaunch(ushort port, string token, IDatabaseFile db) {
EnqueueMessage(new IMessage.StartServer(port, token, db));
}
@@ -113,11 +110,11 @@ public static class ServerLauncher {
private interface IMessage {
public sealed class StartServer : IMessage {
public int Port { get; }
public ushort Port { get; }
public string Token { get; }
public IDatabaseFile Db { get; }
public StartServer(int port, string token, IDatabaseFile db) {
public StartServer(ushort port, string token, IDatabaseFile db) {
this.Port = port;
this.Token = token;
this.Db = db;

View File

@@ -1,5 +1,3 @@
namespace DHT.Server.Service;
readonly struct ServerParameters {
public string Token { get; init; }
}
sealed record ServerParameters(ushort Port, string Token);

View File

@@ -2,6 +2,7 @@ using System.Diagnostics.CodeAnalysis;
using System.Text.Json.Serialization;
using DHT.Server.Database;
using DHT.Server.Endpoints;
using DHT.Server.Service.Middlewares;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Http.Json;
using Microsoft.Extensions.DependencyInjection;
@@ -15,6 +16,7 @@ sealed class Startup {
"https://ptb.discord.com",
"https://canary.discord.com",
"https://discordapp.com",
"null" // For file:// protocol in the Viewer
};
public void ConfigureServices(IServiceCollection services) {
@@ -24,27 +26,27 @@ sealed class Startup {
services.AddCors(static cors => {
cors.AddDefaultPolicy(static builder => {
builder.WithOrigins(AllowedOrigins).AllowCredentials().AllowAnyMethod().AllowAnyHeader();
builder.WithOrigins(AllowedOrigins).AllowCredentials().AllowAnyMethod().AllowAnyHeader().WithExposedHeaders("X-DHT");
});
});
services.AddRoutingCore();
}
[SuppressMessage("ReSharper", "UnusedMember.Global")]
public void Configure(IApplicationBuilder app, IHostApplicationLifetime lifetime, IDatabaseFile db, ServerParameters parameters) {
app.UseRouting();
app.UseMiddleware<ServerLoggingMiddleware>();
app.UseCors();
app.UseMiddleware<ServerAuthorizationMiddleware>();
app.UseRouting();
app.UseEndpoints(endpoints => {
TrackChannelEndpoint trackChannel = new(db, parameters);
endpoints.MapPost("/track-channel", async context => await trackChannel.HandlePost(context));
TrackUsersEndpoint trackUsers = new(db, parameters);
endpoints.MapPost("/track-users", async context => await trackUsers.HandlePost(context));
TrackMessagesEndpoint trackMessages = new(db, parameters);
endpoints.MapPost("/track-messages", async context => await trackMessages.HandlePost(context));
GetAttachmentEndpoint getAttachment = new(db, parameters);
endpoints.MapGet("/get-attachment/{url}", async context => await getAttachment.HandleGet(context));
endpoints.MapGet("/get-tracking-script", new GetTrackingScriptEndpoint(db, parameters).Handle);
endpoints.MapGet("/get-messages", new GetMessagesEndpoint(db).Handle);
endpoints.MapGet("/get-attachment/{url}", new GetAttachmentEndpoint(db).Handle);
endpoints.MapPost("/track-channel", new TrackChannelEndpoint(db).Handle);
endpoints.MapPost("/track-users", new TrackUsersEndpoint(db).Handle);
endpoints.MapPost("/track-messages", new TrackMessagesEndpoint(db).Handle);
});
}
}

View File

@@ -1,3 +1,5 @@
using System.Text;
using System.Text.Json.Serialization.Metadata;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
@@ -12,15 +14,29 @@ public static class HttpOutput {
}
}
public sealed class Json : IHttpOutput {
private readonly object? obj;
public sealed class Text : IHttpOutput {
private readonly string text;
public Json(object? obj) {
this.obj = obj;
public Text(string text) {
this.text = text;
}
public Task WriteTo(HttpResponse response) {
return response.WriteAsJsonAsync(obj);
return response.WriteAsync(text, Encoding.UTF8);
}
}
public sealed class Json<TValue> : IHttpOutput {
private readonly TValue value;
private readonly JsonTypeInfo<TValue> typeInfo;
public Json(TValue value, JsonTypeInfo<TValue> typeInfo) {
this.value = value;
this.typeInfo = typeInfo;
}
public Task WriteTo(HttpResponse response) {
return response.WriteAsJsonAsync(value, typeInfo);
}
}

View File

@@ -0,0 +1,8 @@
using System.Text.Json;
using System.Text.Json.Serialization;
namespace DHT.Utils.Http;
[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase, GenerationMode = JsonSourceGenerationMode.Default)]
[JsonSerializable(typeof(JsonElement))]
public sealed partial class JsonElementContext : JsonSerializerContext {}

View File

@@ -0,0 +1,13 @@
using System.Runtime.InteropServices;
using System.Runtime.Versioning;
namespace DHT.Utils.Logging;
[SupportedOSPlatform("windows")]
public static partial class WindowsConsole {
[LibraryImport("kernel32.dll", SetLastError = true)]
public static partial void AllocConsole();
[LibraryImport("kernel32.dll", SetLastError = true)]
public static partial void FreeConsole();
}

View File

@@ -1,27 +1,25 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<RootNamespace>DHT.Utils</RootNamespace>
<Nullable>enable</Nullable>
<AssemblyName>DiscordHistoryTracker.Utils</AssemblyName>
<PackageId>DiscordHistoryTrackerUtils</PackageId>
<Authors>chylex</Authors>
<Company>DiscordHistoryTracker</Company>
<Product>DiscordHistoryTrackerUtils</Product>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<DebugSymbols>true</DebugSymbols>
<DebugType>none</DebugType>
<PropertyGroup>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="JetBrains.Annotations" Version="2023.2.0" />
</ItemGroup>
<ItemGroup>
<Compile Include="..\Version.cs" Link="Version.cs" />
</ItemGroup>
</Project>

View File

@@ -8,5 +8,5 @@ using DHT.Utils;
namespace DHT.Utils;
static class Version {
public const string Tag = "38.0.0.0";
public const string Tag = "39.1.0.0";
}

View File

@@ -4,11 +4,11 @@ set list=win-x64 linux-x64 osx-x64
rmdir /S /Q bin
(for %%a in (%list%) do (
dotnet publish Desktop -c Release -r %%a -o ./bin/%%a -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true -p:PublishReadyToRun=false -p:PublishTrimmed=true -p:TrimMode=partial -p:JsonSerializerIsReflectionEnabledByDefault=true --self-contained true
dotnet publish Desktop -c Release -r %%a -o ./bin/%%a --self-contained true
powershell "Compress-Archive -Path ./bin/%%a/* -DestinationPath ./bin/%%a.zip -CompressionLevel Optimal"
))
dotnet publish Desktop -c Release -o ./bin/portable --self-contained false
dotnet publish Desktop -c Release -o ./bin/portable -p:PublishSingleFile=false -p:PublishTrimmed=false --self-contained false
powershell "Compress-Archive -Path ./bin/portable/* -DestinationPath ./bin/portable.zip -CompressionLevel Optimal"
echo Done

View File

@@ -17,9 +17,9 @@ rm -rf "./bin"
configurations=(win-x64 linux-x64 osx-x64)
for cfg in ${configurations[@]}; do
dotnet publish Desktop -c Release -r "$cfg" -o "./bin/$cfg" -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true -p:PublishReadyToRun=false -p:PublishTrimmed=true -p:TrimMode=partial -p:JsonSerializerIsReflectionEnabledByDefault=true --self-contained true
dotnet publish Desktop -c Release -r "$cfg" -o "./bin/$cfg" --self-contained true
makezip "$cfg"
done
dotnet publish Desktop -c Release -o "./bin/portable" --self-contained false
dotnet publish Desktop -c Release -o "./bin/portable" -p:PublishSingleFile=false -p:PublishTrimmed=false --self-contained false
makezip "portable"

Binary file not shown.

File diff suppressed because it is too large Load Diff

Binary file not shown.

22
lib/node_modules/commander/LICENSE generated vendored
View File

@@ -1,22 +0,0 @@
(The MIT License)
Copyright (c) 2011 TJ Holowaychuk <tj@vision-media.ca>
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

1110
lib/node_modules/commander/index.js generated vendored

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More