mirror of
https://github.com/chylex/Discord-History-Tracker.git
synced 2025-08-17 19:31:42 +02:00
Compare commits
30 Commits
v38
...
wip-viewer
Author | SHA1 | Date | |
---|---|---|---|
b660af4be0
|
|||
3d9d6a454a
|
|||
ee39780928
|
|||
7b58f973a0
|
|||
93fe018343
|
|||
4f5e27f651
|
|||
cbf81ec95a
|
|||
8a80cb8c20
|
|||
865deb356a
|
|||
069ab97196
|
|||
caab038eaa
|
|||
fb837374fc
|
|||
65d935cca1
|
|||
6e64c86d7a
|
|||
8aeb590bb3
|
|||
8dc1adc9f0
|
|||
ddf70b02e7
|
|||
ef59fd992e
|
|||
d044627fac
|
|||
a624745602 | |||
6da3c185e5 | |||
d4d14cab97
|
|||
095c9a061a
|
|||
d01f9ed218
|
|||
dd6f121059
|
|||
8bba33d815
|
|||
9eab8ac92a
|
|||
fe588686fc
|
|||
7392987165
|
|||
492dddb35d
|
16
README.md
16
README.md
@@ -16,7 +16,6 @@ Fork the repository and clone it to your computer (if you've never used git, you
|
||||
|
||||
Folder organization:
|
||||
* `app/` contains a Visual Studio solution for the desktop app
|
||||
* `lib/` contains utilities required to build the project
|
||||
* `web/` contains source code of the [official website](https://dht.chylex.com), which can be used as a template when making your own website
|
||||
|
||||
To start editing source code for the desktop app, install the [.NET 8 SDK](https://dotnet.microsoft.com/en-us/download/dotnet/8.0), and then open `app/DiscordHistoryTracker.sln` in [Visual Studio](https://visualstudio.microsoft.com/downloads/) or [Rider](https://www.jetbrains.com/rider/).
|
||||
@@ -29,22 +28,13 @@ To build a `Release` version of the desktop app, follow the instructions for you
|
||||
|
||||
#### Release – Windows (64-bit)
|
||||
|
||||
1. Install [Python 3](https://www.python.org/downloads), and ensure the `python` executable is in your `PATH`
|
||||
2. Install [Powershell 5](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows) or newer (on Windows 10, the included version of Powershell should be enough)
|
||||
|
||||
The `lib/` folder contains an installation of [Node](https://nodejs.org/en) and [uglify-js](https://www.npmjs.com/package/uglify-js), which are used to minify the tracking script. This installation will only work on 64-bit Windows; building on 32-bit Windows is not supported, but you can try.
|
||||
1. Install [Powershell 5](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows) or newer (on Windows 10, the included version of Powershell should be enough)
|
||||
|
||||
Run the `app/build.bat` script, and read the [Distribution](#distribution) section below.
|
||||
|
||||
#### Release – Other Operating Systems
|
||||
|
||||
1. Install [Python 3](https://www.python.org/downloads), and ensure the `python` executable exists and launches Python 3
|
||||
- On Debian and derivatives, you can install `python-is-python3`
|
||||
- On other distributions, you can create a link manually, for ex. `ln -s /usr/bin/python3 /usr/bin/python`
|
||||
- If you don't want `python` to mean Python 3, then edit `Desktop.csproj` and change `python` to `python3`
|
||||
2. Install [Node + npm](https://nodejs.org/en)
|
||||
3. Install [uglify-js](https://www.npmjs.com/package/uglify-js) globally (`npm install -g uglify-js`)
|
||||
4. Install the `zip` package from your repository
|
||||
1. Install the `zip` package from your repository
|
||||
|
||||
Run the `app/build.sh` script, and read the [Distribution](#distribution) section below.
|
||||
|
||||
@@ -52,4 +42,4 @@ Run the `app/build.sh` script, and read the [Distribution](#distribution) sectio
|
||||
|
||||
The mentioned build scripts will prepare `Release` builds ready for distribution. Once the script finishes, the `app/bin` folder will contain self-contained executables for each major operating system, and a portable version that works on all other systems but requires .NET 8 to be installed.
|
||||
|
||||
Note that when building on Windows, the generated `.zip` files for Linux and Mac will not have correct file permissions, so it will not be possible to run them by double-clicking `DiscordHistoryTracker`. I tried using Python to re-create the archives with correct file permissions, but found that Linux `zip` tools could not see them. The only working solution is building the Windows + portable version on Windows, and Linux + Mac version on Linux.
|
||||
Note that when building on Windows, the generated `.zip` files for Linux and Mac will not have correct file permissions, so it will not be possible to run them by double-clicking the executable. Since .NET 8 fixed several issues with publishing Windows executables on Linux, I recommend using Linux to build the app for all operating systems.
|
||||
|
@@ -29,9 +29,6 @@
|
||||
<H2CodeStyleSettings version="6">
|
||||
<option name="USE_GENERIC_STYLE" value="true" />
|
||||
</H2CodeStyleSettings>
|
||||
<H2CodeStyleSettings version="6">
|
||||
<option name="USE_GENERIC_STYLE" value="true" />
|
||||
</H2CodeStyleSettings>
|
||||
<HSQLCodeStyleSettings version="6">
|
||||
<option name="USE_GENERIC_STYLE" value="true" />
|
||||
</HSQLCodeStyleSettings>
|
||||
|
@@ -1,8 +1,8 @@
|
||||
<component name="ProjectRunConfigurationManager">
|
||||
<configuration default="false" name="Desktop" type="DotNetProject" factoryName=".NET Project">
|
||||
<option name="EXE_PATH" value="$PROJECT_DIR$/Desktop/bin/Debug/net5.0/DiscordHistoryTracker.exe" />
|
||||
<option name="EXE_PATH" value="$PROJECT_DIR$/.artifacts/bin/Desktop/debug/DiscordHistoryTracker.exe" />
|
||||
<option name="PROGRAM_PARAMETERS" value="" />
|
||||
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$/Desktop/bin/Debug/net5.0" />
|
||||
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$/.artifacts/bin/Desktop/debug" />
|
||||
<option name="PASS_PARENT_ENVS" value="1" />
|
||||
<option name="USE_EXTERNAL_CONSOLE" value="0" />
|
||||
<option name="USE_MONO" value="0" />
|
||||
@@ -12,7 +12,7 @@
|
||||
<option name="PROJECT_ARGUMENTS_TRACKING" value="1" />
|
||||
<option name="PROJECT_WORKING_DIRECTORY_TRACKING" value="1" />
|
||||
<option name="PROJECT_KIND" value="DotNetCore" />
|
||||
<option name="PROJECT_TFM" value="net5.0" />
|
||||
<option name="PROJECT_TFM" value="net8.0" />
|
||||
<method v="2">
|
||||
<option name="Build" />
|
||||
</method>
|
||||
|
@@ -1,23 +0,0 @@
|
||||
<component name="ProjectRunConfigurationManager">
|
||||
<configuration default="false" name="Minify" type="PythonConfigurationType" factoryName="Python">
|
||||
<module name="rider.module" />
|
||||
<option name="INTERPRETER_OPTIONS" value="" />
|
||||
<option name="PARENT_ENVS" value="true" />
|
||||
<envs>
|
||||
<env name="PYTHONUNBUFFERED" value="1" />
|
||||
</envs>
|
||||
<option name="SDK_HOME" value="" />
|
||||
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$" />
|
||||
<option name="IS_MODULE_SDK" value="false" />
|
||||
<option name="ADD_CONTENT_ROOTS" value="false" />
|
||||
<option name="ADD_SOURCE_ROOTS" value="false" />
|
||||
<option name="SCRIPT_NAME" value="$PROJECT_DIR$/minify.py" />
|
||||
<option name="PARAMETERS" value="" />
|
||||
<option name="SHOW_COMMAND_LINE" value="false" />
|
||||
<option name="EMULATE_TERMINAL" value="false" />
|
||||
<option name="MODULE_MODE" value="false" />
|
||||
<option name="REDIRECT_INPUT" value="false" />
|
||||
<option name="INPUT_FILE" value="" />
|
||||
<method v="2" />
|
||||
</configuration>
|
||||
</component>
|
@@ -2,7 +2,8 @@
|
||||
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
|
||||
xmlns:common="clr-namespace:DHT.Desktop.Common"
|
||||
xmlns:system="clr-namespace:System;assembly=System.Runtime"
|
||||
x:Class="DHT.Desktop.App">
|
||||
x:Class="DHT.Desktop.App"
|
||||
RequestedThemeVariant="Light">
|
||||
|
||||
<Application.Styles>
|
||||
|
||||
|
@@ -1,4 +1,3 @@
|
||||
using System;
|
||||
using Avalonia;
|
||||
using Avalonia.Controls.ApplicationLifetimes;
|
||||
using Avalonia.Markup.Xaml;
|
||||
@@ -13,7 +12,7 @@ sealed class App : Application {
|
||||
|
||||
public override void OnFrameworkInitializationCompleted() {
|
||||
if (ApplicationLifetime is IClassicDesktopStyleApplicationLifetime desktop) {
|
||||
desktop.MainWindow = new MainWindow(new Arguments(desktop.Args ?? Array.Empty<string>()));
|
||||
desktop.MainWindow = new MainWindow(Program.Arguments);
|
||||
}
|
||||
|
||||
base.OnFrameworkInitializationCompleted();
|
||||
|
@@ -5,26 +5,33 @@ namespace DHT.Desktop;
|
||||
|
||||
sealed class Arguments {
|
||||
private static readonly Log Log = Log.ForType<Arguments>();
|
||||
|
||||
private const int FirstArgument = 1;
|
||||
|
||||
public static Arguments Empty => new(Array.Empty<string>());
|
||||
|
||||
public bool Console { get; }
|
||||
public string? DatabaseFile { get; }
|
||||
public ushort? ServerPort { get; }
|
||||
public string? ServerToken { get; }
|
||||
|
||||
public Arguments(string[] args) {
|
||||
for (int i = 0; i < args.Length; i++) {
|
||||
for (int i = FirstArgument; i < args.Length; i++) {
|
||||
string key = args[i];
|
||||
|
||||
switch (key) {
|
||||
case "-debug":
|
||||
Log.IsDebugEnabled = true;
|
||||
continue;
|
||||
|
||||
case "-console":
|
||||
Console = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
string value;
|
||||
|
||||
if (i == 0 && !key.StartsWith('-')) {
|
||||
if (i == FirstArgument && !key.StartsWith('-')) {
|
||||
value = key;
|
||||
key = "-db";
|
||||
}
|
||||
|
@@ -1,9 +1,11 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Avalonia.Controls;
|
||||
using Avalonia.Platform.Storage;
|
||||
using Avalonia.Threading;
|
||||
using DHT.Desktop.Dialogs.File;
|
||||
using DHT.Desktop.Dialogs.Message;
|
||||
using DHT.Server.Database;
|
||||
@@ -41,11 +43,16 @@ static class DatabaseGui {
|
||||
});
|
||||
}
|
||||
|
||||
public static async Task<IDatabaseFile?> TryOpenOrCreateDatabaseFromPath(string path, Window window, Func<Task<bool>> checkCanUpgradeDatabase) {
|
||||
public static async Task<IDatabaseFile?> TryOpenOrCreateDatabaseFromPath(string path, Window window, ISchemaUpgradeCallbacks schemaUpgradeCallbacks) {
|
||||
var prevSynchronizationContext = SynchronizationContext.Current;
|
||||
SynchronizationContext.SetSynchronizationContext(new AvaloniaSynchronizationContext());
|
||||
var taskScheduler = TaskScheduler.FromCurrentSynchronizationContext();
|
||||
SynchronizationContext.SetSynchronizationContext(prevSynchronizationContext);
|
||||
|
||||
IDatabaseFile? file = null;
|
||||
|
||||
try {
|
||||
file = await SqliteDatabaseFile.OpenOrCreate(path, checkCanUpgradeDatabase);
|
||||
file = await SqliteDatabaseFile.OpenOrCreate(path, schemaUpgradeCallbacks, taskScheduler);
|
||||
} catch (InvalidDatabaseVersionException ex) {
|
||||
await Dialog.ShowOk(window, "Database Error", "Database '" + Path.GetFileName(path) + "' appears to be corrupted (invalid version: " + ex.Version + ").");
|
||||
} catch (DatabaseTooNewException ex) {
|
||||
|
@@ -1,36 +1,33 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<RootNamespace>DHT.Desktop</RootNamespace>
|
||||
<AssemblyName>DiscordHistoryTracker</AssemblyName>
|
||||
<PackageId>DiscordHistoryTracker</PackageId>
|
||||
</PropertyGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<OutputType>WinExe</OutputType>
|
||||
<Nullable>enable</Nullable>
|
||||
<AssemblyName>DiscordHistoryTracker</AssemblyName>
|
||||
<RootNamespace>DHT.Desktop</RootNamespace>
|
||||
<PackageId>DiscordHistoryTracker</PackageId>
|
||||
<Authors>chylex</Authors>
|
||||
<Company>DiscordHistoryTracker</Company>
|
||||
<Product>DiscordHistoryTracker</Product>
|
||||
<ApplicationIcon>./Resources/icon.ico</ApplicationIcon>
|
||||
<AvaloniaUseCompiledBindingsByDefault>true</AvaloniaUseCompiledBindingsByDefault>
|
||||
<CheckForOverflowUnderflow>true</CheckForOverflowUnderflow>
|
||||
<SatelliteResourceLanguages>en</SatelliteResourceLanguages>
|
||||
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
|
||||
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
|
||||
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
|
||||
</PropertyGroup>
|
||||
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
|
||||
<DebugSymbols>true</DebugSymbols>
|
||||
<DebugType>none</DebugType>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Avalonia" Version="11.0.5" />
|
||||
<PackageReference Include="Avalonia.Controls.DataGrid" Version="11.0.5" />
|
||||
<PackageReference Include="Avalonia.Controls.ItemsRepeater" Version="11.0.5" />
|
||||
<PackageReference Include="Avalonia.Desktop" Version="11.0.5" />
|
||||
<PackageReference Include="Avalonia.Diagnostics" Version="11.0.5" Condition=" '$(Configuration)' == 'Debug' " />
|
||||
<PackageReference Include="Avalonia.Fonts.Inter" Version="11.0.5" />
|
||||
<PackageReference Include="Avalonia.Themes.Fluent" Version="11.0.5" />
|
||||
<PackageReference Include="Avalonia" Version="11.0.6" />
|
||||
<PackageReference Include="Avalonia.Controls.DataGrid" Version="11.0.6" />
|
||||
<PackageReference Include="Avalonia.Controls.ItemsRepeater" Version="11.0.6" />
|
||||
<PackageReference Include="Avalonia.Desktop" Version="11.0.6" />
|
||||
<PackageReference Include="Avalonia.Diagnostics" Version="11.0.6" Condition=" '$(Configuration)' == 'Debug' " />
|
||||
<PackageReference Include="Avalonia.Fonts.Inter" Version="11.0.6" />
|
||||
<PackageReference Include="Avalonia.Themes.Fluent" Version="11.0.6" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\Server\Server.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<Compile Include="..\Version.cs" Link="Version.cs" />
|
||||
<Compile Update="Dialogs\TextBox\TextBoxDialog.axaml.cs">
|
||||
@@ -38,22 +35,11 @@
|
||||
<SubType>Code</SubType>
|
||||
</Compile>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<AvaloniaResource Include="Resources/icon.ico" />
|
||||
<EmbeddedResource Include="../Resources/Tracker/bootstrap.js">
|
||||
<LogicalName>Tracker\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
|
||||
<Link>Resources/Tracker/%(RecursiveDir)%(Filename)%(Extension)</Link>
|
||||
<Visible>false</Visible>
|
||||
</EmbeddedResource>
|
||||
<EmbeddedResource Include="../Resources/Tracker/scripts/**" Condition=" '$(Configuration)' == 'Debug' ">
|
||||
<LogicalName>Tracker\scripts\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
|
||||
<Link>Resources/Tracker/scripts/%(RecursiveDir)%(Filename)%(Extension)</Link>
|
||||
<Visible>false</Visible>
|
||||
</EmbeddedResource>
|
||||
<EmbeddedResource Include="../Resources/Tracker/styles/**">
|
||||
<LogicalName>Tracker\styles\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
|
||||
<Link>Resources/Tracker/styles/%(RecursiveDir)%(Filename)%(Extension)</Link>
|
||||
<Visible>false</Visible>
|
||||
<EmbeddedResource Include="Resources/tracker-loader.js">
|
||||
<LogicalName>tracker-loader.js</LogicalName>
|
||||
</EmbeddedResource>
|
||||
<EmbeddedResource Include="../Resources/Viewer/**">
|
||||
<LogicalName>Viewer\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
|
||||
@@ -61,19 +47,5 @@
|
||||
<Visible>false</Visible>
|
||||
</EmbeddedResource>
|
||||
</ItemGroup>
|
||||
<Target Name="MinifyResources" BeforeTargets="PrepareForBuild" Condition=" '$(Configuration)' == 'Release' ">
|
||||
<PropertyGroup>
|
||||
<MinifiedResourceDir>$(ProjectDir)bin/.res/scripts</MinifiedResourceDir>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<UpToDateCheckInput Include="$(ProjectDir)../Resources/Tracker/scripts/**" Visible="false" />
|
||||
<EmbeddedResource Include="$(MinifiedResourceDir)/discord.js" LogicalName="Tracker\scripts\discord.js" Visible="false" />
|
||||
<EmbeddedResource Include="$(MinifiedResourceDir)/dom.js" LogicalName="Tracker\scripts\dom.js" Visible="false" />
|
||||
<EmbeddedResource Include="$(MinifiedResourceDir)/gui.js" LogicalName="Tracker\scripts\gui.js" Visible="false" />
|
||||
<EmbeddedResource Include="$(MinifiedResourceDir)/settings.js" LogicalName="Tracker\scripts\settings.js" Visible="false" />
|
||||
<EmbeddedResource Include="$(MinifiedResourceDir)/state.js" LogicalName="Tracker\scripts\state.js" Visible="false" />
|
||||
</ItemGroup>
|
||||
<RemoveDir Directories="$(ProjectDir)bin/.res/scripts" />
|
||||
<Exec Command="python $(ProjectDir)../Resources/minify.py" WorkingDirectory="$(ProjectDir)../Resources" IgnoreExitCode="false" />
|
||||
</Target>
|
||||
|
||||
</Project>
|
||||
|
@@ -5,6 +5,7 @@
|
||||
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.CheckBox"
|
||||
mc:Ignorable="d" d:DesignWidth="500"
|
||||
x:Class="DHT.Desktop.Dialogs.CheckBox.CheckBoxDialog"
|
||||
x:DataType="namespace:CheckBoxDialogModel"
|
||||
Title="{Binding Title}"
|
||||
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
|
||||
Width="500" SizeToContent="Height" CanResize="False"
|
||||
|
@@ -5,6 +5,7 @@
|
||||
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.Message"
|
||||
mc:Ignorable="d" d:DesignWidth="500"
|
||||
x:Class="DHT.Desktop.Dialogs.Message.MessageDialog"
|
||||
x:DataType="namespace:MessageDialogModel"
|
||||
Title="{Binding Title}"
|
||||
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
|
||||
Width="500" SizeToContent="Height" CanResize="False"
|
||||
|
@@ -4,4 +4,5 @@ namespace DHT.Desktop.Dialogs.Progress;
|
||||
|
||||
interface IProgressCallback {
|
||||
Task Update(string message, int finishedItems, int totalItems);
|
||||
Task Hide();
|
||||
}
|
||||
|
@@ -5,6 +5,7 @@
|
||||
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.Progress"
|
||||
mc:Ignorable="d" d:DesignWidth="500"
|
||||
x:Class="DHT.Desktop.Dialogs.Progress.ProgressDialog"
|
||||
x:DataType="namespace:ProgressDialogModel"
|
||||
Title="{Binding Title}"
|
||||
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
|
||||
Opened="OnOpened"
|
||||
@@ -31,12 +32,18 @@
|
||||
</Style>
|
||||
</Window.Styles>
|
||||
|
||||
<StackPanel Margin="20">
|
||||
<DockPanel>
|
||||
<TextBlock DockPanel.Dock="Right" Text="{Binding Items}" Classes="items" />
|
||||
<TextBlock DockPanel.Dock="Left" Text="{Binding Message}" />
|
||||
</DockPanel>
|
||||
<ProgressBar Value="{Binding Progress}" />
|
||||
</StackPanel>
|
||||
<ItemsRepeater ItemsSource="{Binding Items}" Margin="0 10">
|
||||
<ItemsRepeater.ItemTemplate>
|
||||
<DataTemplate>
|
||||
<StackPanel Margin="20 10" IsHitTestVisible="{Binding IsVisible}" Opacity="{Binding Opacity}">
|
||||
<DockPanel>
|
||||
<TextBlock DockPanel.Dock="Right" Text="{Binding Items}" Classes="items" />
|
||||
<TextBlock DockPanel.Dock="Left" Text="{Binding Message}" />
|
||||
</DockPanel>
|
||||
<ProgressBar Value="{Binding Progress}" />
|
||||
</StackPanel>
|
||||
</DataTemplate>
|
||||
</ItemsRepeater.ItemTemplate>
|
||||
</ItemsRepeater>
|
||||
|
||||
</Window>
|
||||
|
@@ -8,6 +8,7 @@ namespace DHT.Desktop.Dialogs.Progress;
|
||||
[SuppressMessage("ReSharper", "MemberCanBeInternal")]
|
||||
public sealed partial class ProgressDialog : Window {
|
||||
private bool isFinished = false;
|
||||
private Task progressTask = Task.CompletedTask;
|
||||
|
||||
public ProgressDialog() {
|
||||
InitializeComponent();
|
||||
@@ -15,7 +16,8 @@ public sealed partial class ProgressDialog : Window {
|
||||
|
||||
public void OnOpened(object? sender, EventArgs e) {
|
||||
if (DataContext is ProgressDialogModel model) {
|
||||
Task.Run(model.StartTask).ContinueWith(OnFinished, TaskScheduler.FromCurrentSynchronizationContext());
|
||||
progressTask = Task.Run(model.StartTask);
|
||||
progressTask.ContinueWith(OnFinished, TaskScheduler.FromCurrentSynchronizationContext());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -27,4 +29,9 @@ public sealed partial class ProgressDialog : Window {
|
||||
isFinished = true;
|
||||
Close();
|
||||
}
|
||||
|
||||
public async Task ShowProgressDialog(Window owner) {
|
||||
await ShowDialog(owner);
|
||||
await progressTask;
|
||||
}
|
||||
}
|
||||
|
@@ -1,4 +1,6 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Threading.Tasks;
|
||||
using Avalonia.Threading;
|
||||
using DHT.Desktop.Common;
|
||||
@@ -9,57 +11,43 @@ namespace DHT.Desktop.Dialogs.Progress;
|
||||
sealed class ProgressDialogModel : BaseModel {
|
||||
public string Title { get; init; } = "";
|
||||
|
||||
private string message = "";
|
||||
|
||||
public string Message {
|
||||
get => message;
|
||||
private set => Change(ref message, value);
|
||||
}
|
||||
|
||||
private string items = "";
|
||||
|
||||
public string Items {
|
||||
get => items;
|
||||
private set => Change(ref items, value);
|
||||
}
|
||||
|
||||
private int progress = 0;
|
||||
|
||||
public int Progress {
|
||||
get => progress;
|
||||
private set => Change(ref progress, value);
|
||||
}
|
||||
public IReadOnlyList<ProgressItem> Items { get; } = Array.Empty<ProgressItem>();
|
||||
|
||||
private readonly TaskRunner? task;
|
||||
|
||||
[Obsolete("Designer")]
|
||||
public ProgressDialogModel() {}
|
||||
|
||||
public ProgressDialogModel(TaskRunner task) {
|
||||
public ProgressDialogModel(TaskRunner task, int progressItems = 1) {
|
||||
this.Items = Enumerable.Range(0, progressItems).Select(static _ => new ProgressItem()).ToArray();
|
||||
this.task = task;
|
||||
}
|
||||
|
||||
internal async Task StartTask() {
|
||||
if (task != null) {
|
||||
await task(new Callback(this));
|
||||
await task(Items.Select(static item => new Callback(item)).ToArray());
|
||||
}
|
||||
}
|
||||
|
||||
public delegate Task TaskRunner(IProgressCallback callback);
|
||||
public delegate Task TaskRunner(IReadOnlyList<IProgressCallback> callbacks);
|
||||
|
||||
private sealed class Callback : IProgressCallback {
|
||||
private readonly ProgressDialogModel model;
|
||||
private readonly ProgressItem item;
|
||||
|
||||
public Callback(ProgressDialogModel model) {
|
||||
this.model = model;
|
||||
public Callback(ProgressItem item) {
|
||||
this.item = item;
|
||||
}
|
||||
|
||||
async Task IProgressCallback.Update(string message, int finishedItems, int totalItems) {
|
||||
public async Task Update(string message, int finishedItems, int totalItems) {
|
||||
await Dispatcher.UIThread.InvokeAsync(() => {
|
||||
model.Message = message;
|
||||
model.Items = finishedItems.Format() + " / " + totalItems.Format();
|
||||
model.Progress = 100 * finishedItems / totalItems;
|
||||
item.Message = message;
|
||||
item.Items = totalItems == 0 ? string.Empty : finishedItems.Format() + " / " + totalItems.Format();
|
||||
item.Progress = totalItems == 0 ? 0 : 100 * finishedItems / totalItems;
|
||||
});
|
||||
}
|
||||
|
||||
public Task Hide() {
|
||||
return Update(string.Empty, 0, 0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
41
app/Desktop/Dialogs/Progress/ProgressItem.cs
Normal file
41
app/Desktop/Dialogs/Progress/ProgressItem.cs
Normal file
@@ -0,0 +1,41 @@
|
||||
using DHT.Utils.Models;
|
||||
|
||||
namespace DHT.Desktop.Dialogs.Progress;
|
||||
|
||||
sealed class ProgressItem : BaseModel {
|
||||
private bool isVisible = false;
|
||||
|
||||
public bool IsVisible {
|
||||
get => isVisible;
|
||||
private set {
|
||||
Change(ref isVisible, value);
|
||||
OnPropertyChanged(nameof(Opacity));
|
||||
}
|
||||
}
|
||||
|
||||
public double Opacity => IsVisible ? 1.0 : 0.0;
|
||||
|
||||
private string message = "";
|
||||
|
||||
public string Message {
|
||||
get => message;
|
||||
set {
|
||||
Change(ref message, value);
|
||||
IsVisible = !string.IsNullOrEmpty(value);
|
||||
}
|
||||
}
|
||||
|
||||
private string items = "";
|
||||
|
||||
public string Items {
|
||||
get => items;
|
||||
set => Change(ref items, value);
|
||||
}
|
||||
|
||||
private int progress = 0;
|
||||
|
||||
public int Progress {
|
||||
get => progress;
|
||||
set => Change(ref progress, value);
|
||||
}
|
||||
}
|
@@ -5,6 +5,7 @@
|
||||
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.TextBox"
|
||||
mc:Ignorable="d" d:DesignWidth="500"
|
||||
x:Class="DHT.Desktop.Dialogs.TextBox.TextBoxDialog"
|
||||
x:DataType="namespace:TextBoxDialogModel"
|
||||
Title="{Binding Title}"
|
||||
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
|
||||
Width="500" SizeToContent="Height" CanResize="False"
|
||||
|
@@ -1,9 +1,9 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics.CodeAnalysis;
|
||||
using System.IO;
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Nodes;
|
||||
using System.Threading.Tasks;
|
||||
using DHT.Utils.Logging;
|
||||
using static System.Environment.SpecialFolder;
|
||||
@@ -47,12 +47,12 @@ static class DiscordAppSettings {
|
||||
}
|
||||
}
|
||||
|
||||
private static bool AreDevToolsEnabled(Dictionary<string, object?> json) {
|
||||
return json.TryGetValue(JsonKeyDevTools, out var value) && value is JsonElement { ValueKind: JsonValueKind.True };
|
||||
private static bool AreDevToolsEnabled(JsonObject json) {
|
||||
return json.TryGetPropertyValue(JsonKeyDevTools, out var node) && node?.GetValueKind() == JsonValueKind.True;
|
||||
}
|
||||
|
||||
public static async Task<SettingsJsonResult> ConfigureDevTools(bool enable) {
|
||||
Dictionary<string, object?> json;
|
||||
JsonObject json;
|
||||
|
||||
try {
|
||||
json = await ReadSettingsJson();
|
||||
@@ -109,13 +109,13 @@ static class DiscordAppSettings {
|
||||
return SettingsJsonResult.Success;
|
||||
}
|
||||
|
||||
private static async Task<Dictionary<string, object?>> ReadSettingsJson() {
|
||||
private static async Task<JsonObject> ReadSettingsJson() {
|
||||
await using var stream = new FileStream(JsonFilePath, FileMode.Open, FileAccess.Read, FileShare.Read);
|
||||
return await JsonSerializer.DeserializeAsync<Dictionary<string, object?>?>(stream) ?? throw new JsonException();
|
||||
return await JsonSerializer.DeserializeAsync(stream, DiscordAppSettingsJsonContext.Default.JsonObject) ?? throw new JsonException();
|
||||
}
|
||||
|
||||
private static async Task WriteSettingsJson(Dictionary<string, object?> json) {
|
||||
private static async Task WriteSettingsJson(JsonObject json) {
|
||||
await using var stream = new FileStream(JsonFilePath, FileMode.Truncate, FileAccess.Write, FileShare.None);
|
||||
await JsonSerializer.SerializeAsync(stream, json, new JsonSerializerOptions { WriteIndented = true });
|
||||
await JsonSerializer.SerializeAsync(stream, json, DiscordAppSettingsJsonContext.Default.JsonObject);
|
||||
}
|
||||
}
|
||||
|
8
app/Desktop/Discord/DiscordAppSettingsJsonContext.cs
Normal file
8
app/Desktop/Discord/DiscordAppSettingsJsonContext.cs
Normal file
@@ -0,0 +1,8 @@
|
||||
using System.Text.Json.Nodes;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace DHT.Desktop.Discord;
|
||||
|
||||
[JsonSourceGenerationOptions(GenerationMode = JsonSourceGenerationMode.Default, WriteIndented = true)]
|
||||
[JsonSerializable(typeof(JsonObject))]
|
||||
sealed partial class DiscordAppSettingsJsonContext : JsonSerializerContext {}
|
@@ -5,6 +5,7 @@
|
||||
xmlns:main="clr-namespace:DHT.Desktop.Main"
|
||||
mc:Ignorable="d" d:DesignWidth="480" d:DesignHeight="295"
|
||||
x:Class="DHT.Desktop.Main.AboutWindow"
|
||||
x:DataType="main:AboutWindowModel"
|
||||
Title="About Discord History Tracker"
|
||||
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
|
||||
Width="480" Height="295" CanResize="False"
|
||||
|
@@ -4,7 +4,8 @@
|
||||
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
|
||||
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
|
||||
mc:Ignorable="d"
|
||||
x:Class="DHT.Desktop.Main.Controls.AttachmentFilterPanel">
|
||||
x:Class="DHT.Desktop.Main.Controls.AttachmentFilterPanel"
|
||||
x:DataType="controls:AttachmentFilterPanelModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<controls:AttachmentFilterPanelModel />
|
||||
|
@@ -4,7 +4,8 @@
|
||||
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
|
||||
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
|
||||
mc:Ignorable="d"
|
||||
x:Class="DHT.Desktop.Main.Controls.MessageFilterPanel">
|
||||
x:Class="DHT.Desktop.Main.Controls.MessageFilterPanel"
|
||||
x:DataType="controls:MessageFilterPanelModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<controls:MessageFilterPanelModel />
|
||||
|
@@ -4,7 +4,8 @@
|
||||
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
|
||||
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
|
||||
mc:Ignorable="d"
|
||||
x:Class="DHT.Desktop.Main.Controls.ServerConfigurationPanel">
|
||||
x:Class="DHT.Desktop.Main.Controls.ServerConfigurationPanel"
|
||||
x:DataType="controls:ServerConfigurationPanelModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<controls:ServerConfigurationPanelModel />
|
||||
|
@@ -4,7 +4,8 @@
|
||||
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
|
||||
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
|
||||
mc:Ignorable="d"
|
||||
x:Class="DHT.Desktop.Main.Controls.StatusBar">
|
||||
x:Class="DHT.Desktop.Main.Controls.StatusBar"
|
||||
x:DataType="controls:StatusBarModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<controls:StatusBarModel />
|
||||
|
@@ -5,6 +5,7 @@
|
||||
xmlns:main="clr-namespace:DHT.Desktop.Main"
|
||||
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
|
||||
x:Class="DHT.Desktop.Main.MainWindow"
|
||||
x:DataType="main:MainWindowModel"
|
||||
Title="{Binding Title}"
|
||||
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
|
||||
Width="800" Height="500"
|
||||
|
@@ -5,7 +5,8 @@
|
||||
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
|
||||
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
|
||||
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
|
||||
x:Class="DHT.Desktop.Main.Pages.AdvancedPage">
|
||||
x:Class="DHT.Desktop.Main.Pages.AdvancedPage"
|
||||
x:DataType="pages:AdvancedPageModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<pages:AdvancedPageModel />
|
||||
|
@@ -5,7 +5,8 @@
|
||||
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
|
||||
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
|
||||
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
|
||||
x:Class="DHT.Desktop.Main.Pages.AttachmentsPage">
|
||||
x:Class="DHT.Desktop.Main.Pages.AttachmentsPage"
|
||||
x:DataType="pages:AttachmentsPageModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<pages:AttachmentsPageModel />
|
||||
@@ -35,7 +36,7 @@
|
||||
<TextBlock Text="{Binding DownloadMessage}" Margin="10 0 0 0" VerticalAlignment="Center" DockPanel.Dock="Left" />
|
||||
<ProgressBar Value="{Binding DownloadProgress}" IsVisible="{Binding IsDownloading}" Margin="15 0" VerticalAlignment="Center" DockPanel.Dock="Right" />
|
||||
</DockPanel>
|
||||
<controls:AttachmentFilterPanel DataContext="{Binding FilterModel}" IsEnabled="{Binding !DataContext.IsDownloading, RelativeSource={RelativeSource AncestorType=UserControl}}" />
|
||||
<controls:AttachmentFilterPanel DataContext="{Binding FilterModel}" IsEnabled="{Binding !IsDownloading, RelativeSource={RelativeSource AncestorType=pages:AttachmentsPageModel}}" />
|
||||
<StackPanel Orientation="Vertical" Spacing="12">
|
||||
<Expander Header="Download Status" IsExpanded="True">
|
||||
<DataGrid ItemsSource="{Binding StatisticsRows}" AutoGenerateColumns="False" CanUserReorderColumns="False" CanUserResizeColumns="False" CanUserSortColumns="False" IsReadOnly="True">
|
||||
|
@@ -4,7 +4,8 @@
|
||||
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
|
||||
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
|
||||
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
|
||||
x:Class="DHT.Desktop.Main.Pages.DatabasePage">
|
||||
x:Class="DHT.Desktop.Main.Pages.DatabasePage"
|
||||
x:DataType="pages:DatabasePageModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<pages:DatabasePageModel />
|
||||
|
@@ -17,6 +17,7 @@ using DHT.Desktop.Dialogs.TextBox;
|
||||
using DHT.Server.Data;
|
||||
using DHT.Server.Database;
|
||||
using DHT.Server.Database.Import;
|
||||
using DHT.Server.Database.Sqlite;
|
||||
using DHT.Utils.Logging;
|
||||
using DHT.Utils.Models;
|
||||
|
||||
@@ -77,31 +78,18 @@ sealed class DatabasePageModel : BaseModel {
|
||||
}
|
||||
|
||||
ProgressDialog progressDialog = new ProgressDialog();
|
||||
progressDialog.DataContext = new ProgressDialogModel(async callback => await MergeWithDatabaseFromPaths(Db, paths, progressDialog, callback)) {
|
||||
progressDialog.DataContext = new ProgressDialogModel(async callbacks => await MergeWithDatabaseFromPaths(Db, paths, progressDialog, callbacks[0])) {
|
||||
Title = "Database Merge"
|
||||
};
|
||||
|
||||
await progressDialog.ShowDialog(window);
|
||||
await progressDialog.ShowProgressDialog(window);
|
||||
}
|
||||
|
||||
private static async Task MergeWithDatabaseFromPaths(IDatabaseFile target, string[] paths, ProgressDialog dialog, IProgressCallback callback) {
|
||||
int total = paths.Length;
|
||||
|
||||
DialogResult.YesNo? upgradeResult = null;
|
||||
|
||||
async Task<bool> CheckCanUpgradeDatabase() {
|
||||
upgradeResult ??= total > 1
|
||||
? await DatabaseGui.ShowCanUpgradeMultipleDatabaseDialog(dialog)
|
||||
: await DatabaseGui.ShowCanUpgradeDatabaseDialog(dialog);
|
||||
|
||||
return DialogResult.YesNo.Yes == upgradeResult;
|
||||
}
|
||||
|
||||
var schemaUpgradeCallbacks = new SchemaUpgradeCallbacks(dialog, paths.Length);
|
||||
|
||||
await PerformImport(target, paths, dialog, callback, "Database Merge", "Database Error", "database file", async path => {
|
||||
SynchronizationContext? prevSyncContext = SynchronizationContext.Current;
|
||||
SynchronizationContext.SetSynchronizationContext(new AvaloniaSynchronizationContext());
|
||||
IDatabaseFile? db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, dialog, CheckCanUpgradeDatabase);
|
||||
SynchronizationContext.SetSynchronizationContext(prevSyncContext);
|
||||
IDatabaseFile? db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, dialog, schemaUpgradeCallbacks);
|
||||
|
||||
if (db == null) {
|
||||
return false;
|
||||
@@ -116,6 +104,41 @@ sealed class DatabasePageModel : BaseModel {
|
||||
});
|
||||
}
|
||||
|
||||
private sealed class SchemaUpgradeCallbacks : ISchemaUpgradeCallbacks {
|
||||
private readonly ProgressDialog dialog;
|
||||
private readonly int total;
|
||||
private bool? decision;
|
||||
|
||||
public SchemaUpgradeCallbacks(ProgressDialog dialog, int total) {
|
||||
this.total = total;
|
||||
this.dialog = dialog;
|
||||
}
|
||||
|
||||
public async Task<bool> CanUpgrade() {
|
||||
return decision ??= (total > 1
|
||||
? await DatabaseGui.ShowCanUpgradeMultipleDatabaseDialog(dialog)
|
||||
: await DatabaseGui.ShowCanUpgradeDatabaseDialog(dialog)) == DialogResult.YesNo.Yes;
|
||||
}
|
||||
|
||||
public Task Start(int versionSteps, Func<ISchemaUpgradeCallbacks.IProgressReporter, Task> doUpgrade) {
|
||||
return doUpgrade(new NullReporter());
|
||||
}
|
||||
|
||||
private sealed class NullReporter : ISchemaUpgradeCallbacks.IProgressReporter {
|
||||
public Task NextVersion() {
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task MainWork(string message, int finishedItems, int totalItems) {
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task SubWork(string message, int finishedItems, int totalItems) {
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public async void ImportLegacyArchive() {
|
||||
var paths = await window.StorageProvider.OpenFiles(new FilePickerOpenOptions {
|
||||
Title = "Open Legacy DHT Archive",
|
||||
@@ -128,11 +151,11 @@ sealed class DatabasePageModel : BaseModel {
|
||||
}
|
||||
|
||||
ProgressDialog progressDialog = new ProgressDialog();
|
||||
progressDialog.DataContext = new ProgressDialogModel(async callback => await ImportLegacyArchiveFromPaths(Db, paths, progressDialog, callback)) {
|
||||
progressDialog.DataContext = new ProgressDialogModel(async callbacks => await ImportLegacyArchiveFromPaths(Db, paths, progressDialog, callbacks[0])) {
|
||||
Title = "Legacy Archive Import"
|
||||
};
|
||||
|
||||
await progressDialog.ShowDialog(window);
|
||||
await progressDialog.ShowProgressDialog(window);
|
||||
}
|
||||
|
||||
private static async Task ImportLegacyArchiveFromPaths(IDatabaseFile target, string[] paths, ProgressDialog dialog, IProgressCallback callback) {
|
||||
|
@@ -4,7 +4,8 @@
|
||||
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
|
||||
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
|
||||
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
|
||||
x:Class="DHT.Desktop.Main.Pages.DebugPage">
|
||||
x:Class="DHT.Desktop.Main.Pages.DebugPage"
|
||||
x:DataType="pages:DebugPageModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<pages:DebugPageModel />
|
||||
|
@@ -45,12 +45,12 @@ namespace DHT.Desktop.Main.Pages {
|
||||
}
|
||||
|
||||
ProgressDialog progressDialog = new ProgressDialog {
|
||||
DataContext = new ProgressDialogModel(async callback => await GenerateRandomData(channels, users, messages, callback)) {
|
||||
DataContext = new ProgressDialogModel(async callbacks => await GenerateRandomData(channels, users, messages, callbacks[0])) {
|
||||
Title = "Generating Random Data"
|
||||
}
|
||||
};
|
||||
|
||||
await progressDialog.ShowDialog(window);
|
||||
await progressDialog.ShowProgressDialog(window);
|
||||
}
|
||||
|
||||
private const int BatchSize = 500;
|
||||
|
@@ -4,7 +4,8 @@
|
||||
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
|
||||
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
|
||||
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
|
||||
x:Class="DHT.Desktop.Main.Pages.TrackingPage">
|
||||
x:Class="DHT.Desktop.Main.Pages.TrackingPage"
|
||||
x:DataType="pages:TrackingPageModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<pages:TrackingPageModel />
|
||||
|
@@ -52,14 +52,10 @@ sealed class TrackingPageModel : BaseModel {
|
||||
OnPropertyChanged(nameof(IsToggleAppDevToolsButtonEnabled));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
public async Task<bool> OnClickCopyTrackingScript() {
|
||||
string bootstrap = await Resources.ReadTextAsync("Tracker/bootstrap.js");
|
||||
string script = bootstrap.Replace("= 0; /*[PORT]*/", "= " + ServerManager.Port + ";")
|
||||
.Replace("/*[TOKEN]*/", HttpUtility.JavaScriptStringEncode(ServerManager.Token))
|
||||
.Replace("/*[IMPORTS]*/", await Resources.ReadJoinedAsync("Tracker/scripts/", '\n'))
|
||||
.Replace("/*[CSS-CONTROLLER]*/", await Resources.ReadTextAsync("Tracker/styles/controller.css"))
|
||||
.Replace("/*[CSS-SETTINGS]*/", await Resources.ReadTextAsync("Tracker/styles/settings.css"));
|
||||
string url = $"http://127.0.0.1:{ServerManager.Port}/get-tracking-script?token={HttpUtility.UrlEncode(ServerManager.Token)}";
|
||||
string script = (await Resources.ReadTextAsync("tracker-loader.js")).Trim().Replace("{url}", url);
|
||||
|
||||
var clipboard = window.Clipboard;
|
||||
if (clipboard == null) {
|
||||
|
@@ -5,7 +5,8 @@
|
||||
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
|
||||
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
|
||||
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
|
||||
x:Class="DHT.Desktop.Main.Pages.ViewerPage">
|
||||
x:Class="DHT.Desktop.Main.Pages.ViewerPage"
|
||||
x:DataType="pages:ViewerPageModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<pages:ViewerPageModel />
|
||||
|
@@ -35,7 +35,7 @@ sealed class ViewerPageModel : BaseModel, IDisposable {
|
||||
set => Change(ref hasFilters, value);
|
||||
}
|
||||
|
||||
private MessageFilterPanelModel FilterModel { get; }
|
||||
public MessageFilterPanelModel FilterModel { get; }
|
||||
|
||||
private readonly Window window;
|
||||
private readonly IDatabaseFile db;
|
||||
@@ -65,6 +65,8 @@ sealed class ViewerPageModel : BaseModel, IDisposable {
|
||||
string indexFile = await Resources.ReadTextAsync("Viewer/index.html");
|
||||
string viewerTemplate = indexFile.Replace("/*[JS]*/", await Resources.ReadJoinedAsync("Viewer/scripts/", '\n'))
|
||||
.Replace("/*[CSS]*/", await Resources.ReadJoinedAsync("Viewer/styles/", '\n'));
|
||||
|
||||
viewerTemplate = strategy.ProcessViewerTemplate(viewerTemplate);
|
||||
|
||||
int viewerArchiveTagStart = viewerTemplate.IndexOf(ArchiveTag);
|
||||
int viewerArchiveTagEnd = viewerArchiveTagStart + ArchiveTag.Length;
|
||||
|
@@ -5,7 +5,8 @@
|
||||
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
|
||||
xmlns:screens="clr-namespace:DHT.Desktop.Main.Screens"
|
||||
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
|
||||
x:Class="DHT.Desktop.Main.Screens.MainContentScreen">
|
||||
x:Class="DHT.Desktop.Main.Screens.MainContentScreen"
|
||||
x:DataType="screens:MainContentScreenModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<screens:MainContentScreenModel />
|
||||
|
@@ -4,7 +4,8 @@
|
||||
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
|
||||
xmlns:screens="clr-namespace:DHT.Desktop.Main.Screens"
|
||||
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
|
||||
x:Class="DHT.Desktop.Main.Screens.WelcomeScreen">
|
||||
x:Class="DHT.Desktop.Main.Screens.WelcomeScreen"
|
||||
x:DataType="screens:WelcomeScreenModel">
|
||||
|
||||
<Design.DataContext>
|
||||
<screens:WelcomeScreenModel />
|
||||
|
@@ -1,10 +1,13 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Threading.Tasks;
|
||||
using Avalonia.Controls;
|
||||
using DHT.Desktop.Common;
|
||||
using DHT.Desktop.Dialogs.Message;
|
||||
using DHT.Desktop.Dialogs.Progress;
|
||||
using DHT.Server.Database;
|
||||
using DHT.Server.Database.Sqlite;
|
||||
using DHT.Utils.Models;
|
||||
|
||||
namespace DHT.Desktop.Main.Screens;
|
||||
@@ -39,14 +42,71 @@ sealed class WelcomeScreenModel : BaseModel, IDisposable {
|
||||
}
|
||||
|
||||
dbFilePath = path;
|
||||
Db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, window, CheckCanUpgradeDatabase);
|
||||
Db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, window, new SchemaUpgradeCallbacks(window));
|
||||
|
||||
OnPropertyChanged(nameof(Db));
|
||||
OnPropertyChanged(nameof(HasDatabase));
|
||||
}
|
||||
|
||||
private async Task<bool> CheckCanUpgradeDatabase() {
|
||||
return DialogResult.YesNo.Yes == await DatabaseGui.ShowCanUpgradeDatabaseDialog(window);
|
||||
private sealed class SchemaUpgradeCallbacks : ISchemaUpgradeCallbacks {
|
||||
private readonly Window window;
|
||||
|
||||
public SchemaUpgradeCallbacks(Window window) {
|
||||
this.window = window;
|
||||
}
|
||||
|
||||
public async Task<bool> CanUpgrade() {
|
||||
return DialogResult.YesNo.Yes == await DatabaseGui.ShowCanUpgradeDatabaseDialog(window);
|
||||
}
|
||||
|
||||
public async Task Start(int versionSteps, Func<ISchemaUpgradeCallbacks.IProgressReporter, Task> doUpgrade) {
|
||||
async Task StartUpgrade(IReadOnlyList<IProgressCallback> callbacks) {
|
||||
var reporter = new ProgressReporter(versionSteps, callbacks);
|
||||
await reporter.NextVersion();
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(800));
|
||||
await doUpgrade(reporter);
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(600));
|
||||
}
|
||||
|
||||
await new ProgressDialog {
|
||||
DataContext = new ProgressDialogModel(StartUpgrade, progressItems: 3) {
|
||||
Title = "Upgrading Database"
|
||||
}
|
||||
}.ShowProgressDialog(window);
|
||||
}
|
||||
|
||||
private sealed class ProgressReporter : ISchemaUpgradeCallbacks.IProgressReporter {
|
||||
private readonly IReadOnlyList<IProgressCallback> callbacks;
|
||||
|
||||
private readonly int versionSteps;
|
||||
private int versionProgress = 0;
|
||||
|
||||
public ProgressReporter(int versionSteps, IReadOnlyList<IProgressCallback> callbacks) {
|
||||
this.callbacks = callbacks;
|
||||
this.versionSteps = versionSteps;
|
||||
}
|
||||
|
||||
public async Task NextVersion() {
|
||||
await callbacks[0].Update("Upgrading schema version...", versionProgress++, versionSteps);
|
||||
await HideChildren(0);
|
||||
}
|
||||
|
||||
public async Task MainWork(string message, int finishedItems, int totalItems) {
|
||||
await callbacks[1].Update(message, finishedItems, totalItems);
|
||||
await HideChildren(1);
|
||||
}
|
||||
|
||||
public async Task SubWork(string message, int finishedItems, int totalItems) {
|
||||
await callbacks[2].Update(message, finishedItems, totalItems);
|
||||
await HideChildren(2);
|
||||
}
|
||||
|
||||
private async Task HideChildren(int parentIndex) {
|
||||
for (int i = parentIndex + 1; i < callbacks.Count; i++) {
|
||||
await callbacks[i].Hide();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public void CloseDatabase() {
|
||||
|
@@ -1,6 +1,8 @@
|
||||
using System.Globalization;
|
||||
using System;
|
||||
using System.Globalization;
|
||||
using System.Reflection;
|
||||
using Avalonia;
|
||||
using DHT.Utils.Logging;
|
||||
using DHT.Utils.Resources;
|
||||
|
||||
namespace DHT.Desktop;
|
||||
@@ -9,6 +11,7 @@ static class Program {
|
||||
public static string Version { get; }
|
||||
public static CultureInfo Culture { get; }
|
||||
public static ResourceLoader Resources { get; }
|
||||
public static Arguments Arguments { get; }
|
||||
|
||||
static Program() {
|
||||
var assembly = Assembly.GetExecutingAssembly();
|
||||
@@ -25,10 +28,21 @@ static class Program {
|
||||
CultureInfo.DefaultThreadCurrentUICulture = CultureInfo.InvariantCulture;
|
||||
|
||||
Resources = new ResourceLoader(assembly);
|
||||
Arguments = new Arguments(Environment.GetCommandLineArgs());
|
||||
}
|
||||
|
||||
public static void Main(string[] args) {
|
||||
BuildAvaloniaApp().StartWithClassicDesktopLifetime(args);
|
||||
if (Arguments.Console && OperatingSystem.IsWindows()) {
|
||||
WindowsConsole.AllocConsole();
|
||||
}
|
||||
|
||||
try {
|
||||
BuildAvaloniaApp().StartWithClassicDesktopLifetime(args);
|
||||
} finally {
|
||||
if (Arguments.Console && OperatingSystem.IsWindows()) {
|
||||
WindowsConsole.FreeConsole();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static AppBuilder BuildAvaloniaApp() {
|
||||
|
1
app/Desktop/Resources/tracker-loader.js
Normal file
1
app/Desktop/Resources/tracker-loader.js
Normal file
@@ -0,0 +1 @@
|
||||
fetch("{url}").then(r => r.ok ? (r.headers.get("X-DHT") === "1" ? r.text() : Promise.reject("Invalid response")) : Promise.reject(r.status + " " + r.statusText)).then(s => eval(s)).catch(e => alert("Could not load tracking script:\n" + e));
|
49
app/Directory.Build.props
Normal file
49
app/Directory.Build.props
Normal file
@@ -0,0 +1,49 @@
|
||||
<Project>
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net8.0</TargetFramework>
|
||||
<LangVersion>11</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
</PropertyGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<Authors>chylex</Authors>
|
||||
<Company>DiscordHistoryTracker</Company>
|
||||
<Product>DiscordHistoryTracker</Product>
|
||||
</PropertyGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
|
||||
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
|
||||
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
|
||||
</PropertyGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<SuppressTrimAnalysisWarnings>false</SuppressTrimAnalysisWarnings>
|
||||
<PublishTrimmed>true</PublishTrimmed>
|
||||
<TrimMode>partial</TrimMode>
|
||||
<EnableUnsafeBinaryFormatterSerialization>false</EnableUnsafeBinaryFormatterSerialization>
|
||||
<EnableUnsafeUTF7Encoding>false</EnableUnsafeUTF7Encoding>
|
||||
<EventSourceSupport>false</EventSourceSupport>
|
||||
<HttpActivityPropagationSupport>false</HttpActivityPropagationSupport>
|
||||
<JsonSerializerIsReflectionEnabledByDefault>false</JsonSerializerIsReflectionEnabledByDefault>
|
||||
</PropertyGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<PublishSingleFile>true</PublishSingleFile>
|
||||
<PublishReadyToRun>false</PublishReadyToRun>
|
||||
<EnableCompressionInSingleFile>true</EnableCompressionInSingleFile>
|
||||
<IncludeNativeLibrariesForSelfExtract>true</IncludeNativeLibrariesForSelfExtract>
|
||||
</PropertyGroup>
|
||||
|
||||
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
|
||||
<DebugSymbols>false</DebugSymbols>
|
||||
<DebugType>none</DebugType>
|
||||
</PropertyGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<UseArtifactsOutput>true</UseArtifactsOutput>
|
||||
<ArtifactsPath>$(MSBuildThisFileDirectory).artifacts</ArtifactsPath>
|
||||
</PropertyGroup>
|
||||
|
||||
</Project>
|
@@ -1,8 +0,0 @@
|
||||
<Project>
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net8.0</TargetFramework>
|
||||
<LangVersion>11</LangVersion>
|
||||
</PropertyGroup>
|
||||
|
||||
</Project>
|
5
app/Resources/Tracker/bootstrap.js
vendored
5
app/Resources/Tracker/bootstrap.js
vendored
@@ -64,9 +64,11 @@
|
||||
let action = null;
|
||||
|
||||
if (!DISCORD.hasMoreMessages()) {
|
||||
console.debug("[DHT] Reached first message.");
|
||||
action = SETTINGS.afterFirstMsg;
|
||||
}
|
||||
if (isNoAction(action) && !anyNewMessages) {
|
||||
console.debug("[DHT] No new messages.");
|
||||
action = SETTINGS.afterSavedMsg;
|
||||
}
|
||||
|
||||
@@ -121,7 +123,7 @@
|
||||
onTrackingContinued(false);
|
||||
}
|
||||
else {
|
||||
const anyNewMessages = await STATE.addDiscordMessages(info.id, messages);
|
||||
const anyNewMessages = await STATE.addDiscordMessages(messages);
|
||||
onTrackingContinued(anyNewMessages);
|
||||
}
|
||||
} catch (e) {
|
||||
@@ -156,3 +158,4 @@
|
||||
GUI.showSettings();
|
||||
}
|
||||
})();
|
||||
/*[DEBUGGER]*/
|
||||
|
@@ -1,5 +1,23 @@
|
||||
// noinspection JSUnresolvedVariable
|
||||
// noinspection LocalVariableNamingConventionJS
|
||||
class DISCORD {
|
||||
|
||||
// https://discord.com/developers/docs/resources/channel#channel-object-channel-types
|
||||
static CHANNEL_TYPE = {
|
||||
DM: 1,
|
||||
GROUP_DM: 3,
|
||||
ANNOUNCEMENT_THREAD: 10,
|
||||
PUBLIC_THREAD: 11,
|
||||
PRIVATE_THREAD: 12
|
||||
};
|
||||
|
||||
// https://discord.com/developers/docs/resources/channel#message-object-message-types
|
||||
static MESSAGE_TYPE = {
|
||||
DEFAULT: 0,
|
||||
REPLY: 19,
|
||||
THREAD_STARTER: 21
|
||||
};
|
||||
|
||||
static getMessageOuterElement() {
|
||||
return DOM.queryReactClass("messagesWrapper");
|
||||
}
|
||||
@@ -28,46 +46,11 @@ class DISCORD {
|
||||
* Calls the provided function with a list of messages whenever the currently loaded messages change.
|
||||
*/
|
||||
static setupMessageCallback(callback) {
|
||||
let skipsLeft = 0;
|
||||
let waitForCleanup = false;
|
||||
const previousMessages = new Set();
|
||||
|
||||
const timer = window.setInterval(() => {
|
||||
if (skipsLeft > 0) {
|
||||
--skipsLeft;
|
||||
return;
|
||||
}
|
||||
|
||||
const view = this.getMessageOuterElement();
|
||||
|
||||
if (!view) {
|
||||
skipsLeft = 2;
|
||||
return;
|
||||
}
|
||||
|
||||
const anyMessage = DOM.queryReactClass("message", this.getMessageOuterElement());
|
||||
const messageCount = anyMessage ? anyMessage.parentElement.children.length : 0;
|
||||
|
||||
if (messageCount > 300) {
|
||||
if (waitForCleanup) {
|
||||
return;
|
||||
}
|
||||
|
||||
skipsLeft = 3;
|
||||
waitForCleanup = true;
|
||||
|
||||
window.setTimeout(() => {
|
||||
const view = this.getMessageScrollerElement();
|
||||
// noinspection JSUnusedGlobalSymbols
|
||||
view.scrollTop = view.scrollHeight / 2;
|
||||
}, 1);
|
||||
}
|
||||
else {
|
||||
waitForCleanup = false;
|
||||
}
|
||||
|
||||
const messages = this.getMessages();
|
||||
const hasChanged = messages.some(message => !previousMessages.has(message.id)) || !this.hasMoreMessages();
|
||||
const onMessageElementsChanged = function() {
|
||||
const messages = DISCORD.getMessages();
|
||||
const hasChanged = messages.some(message => !previousMessages.has(message.id)) || !DISCORD.hasMoreMessages();
|
||||
|
||||
if (!hasChanged) {
|
||||
return;
|
||||
@@ -79,24 +62,74 @@ class DISCORD {
|
||||
}
|
||||
|
||||
callback(messages);
|
||||
}, 200);
|
||||
};
|
||||
|
||||
window.DHT_ON_UNLOAD.push(() => window.clearInterval(timer));
|
||||
let debounceTimer;
|
||||
|
||||
/**
|
||||
* Do not trigger the callback too often due to autoscrolling.
|
||||
*/
|
||||
const onMessageElementsChangedLater = function() {
|
||||
window.clearTimeout(debounceTimer);
|
||||
debounceTimer = window.setTimeout(onMessageElementsChanged, 100);
|
||||
};
|
||||
|
||||
const observer = new MutationObserver(function () {
|
||||
onMessageElementsChangedLater();
|
||||
});
|
||||
|
||||
let skipsLeft = 0;
|
||||
let observedElement = null;
|
||||
|
||||
const observerTimer = window.setInterval(() => {
|
||||
if (skipsLeft > 0) {
|
||||
--skipsLeft;
|
||||
return;
|
||||
}
|
||||
|
||||
const view = this.getMessageOuterElement();
|
||||
|
||||
if (!view) {
|
||||
skipsLeft = 1;
|
||||
return;
|
||||
}
|
||||
|
||||
if (observedElement !== null && observedElement.isConnected) {
|
||||
return;
|
||||
}
|
||||
|
||||
observedElement = view.querySelector("[data-list-id='chat-messages']");
|
||||
|
||||
if (observedElement) {
|
||||
console.debug("[DHT] Observed message container.");
|
||||
observer.observe(observedElement, { childList: true });
|
||||
onMessageElementsChangedLater();
|
||||
}
|
||||
}, 400);
|
||||
|
||||
window.DHT_ON_UNLOAD.push(() => {
|
||||
observer.disconnect();
|
||||
observedElement = null;
|
||||
window.clearInterval(observerTimer);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the property object of a message element.
|
||||
* @returns { null | { message: DiscordMessage, channel: Object } }
|
||||
* Returns the message from a message element.
|
||||
* @returns { null | DiscordMessage } }
|
||||
*/
|
||||
static getMessageElementProps(ele) {
|
||||
static getMessageFromElement(ele) {
|
||||
const props = DOM.getReactProps(ele);
|
||||
|
||||
if (props.children && props.children.length) {
|
||||
for (let i = 3; i < props.children.length; i++) {
|
||||
const childProps = props.children[i].props;
|
||||
if (props && Array.isArray(props.children)) {
|
||||
for (const child of props.children) {
|
||||
if (!(child instanceof Object)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (childProps && "message" in childProps && "channel" in childProps) {
|
||||
return childProps;
|
||||
const childProps = child.props;
|
||||
if (childProps instanceof Object && "message" in childProps) {
|
||||
return childProps.message;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -113,10 +146,10 @@ class DISCORD {
|
||||
|
||||
for (const ele of this.getMessageElements()) {
|
||||
try {
|
||||
const props = this.getMessageElementProps(ele);
|
||||
const message = this.getMessageFromElement(ele);
|
||||
|
||||
if (props != null) {
|
||||
messages.push(props.message);
|
||||
if (message != null) {
|
||||
messages.push(message);
|
||||
}
|
||||
} catch (e) {
|
||||
console.error("[DHT] Error extracing message data, skipping it.", e, ele, DOM.tryGetReactProps(ele));
|
||||
@@ -137,7 +170,7 @@ class DISCORD {
|
||||
*/
|
||||
static getSelectedChannel() {
|
||||
try {
|
||||
let obj;
|
||||
let obj = null;
|
||||
|
||||
try {
|
||||
for (const child of DOM.getReactProps(DOM.queryReactClass("chatContent")).children) {
|
||||
@@ -148,15 +181,6 @@ class DISCORD {
|
||||
}
|
||||
} catch (e) {
|
||||
console.error("[DHT] Error retrieving selected channel from 'chatContent' element.", e);
|
||||
|
||||
for (const ele of this.getMessageElements()) {
|
||||
const props = this.getMessageElementProps(ele);
|
||||
|
||||
if (props != null) {
|
||||
obj = props.channel;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!obj || typeof obj.id !== "string") {
|
||||
@@ -185,8 +209,8 @@ class DISCORD {
|
||||
|
||||
// https://discord.com/developers/docs/resources/channel#channel-object-channel-types
|
||||
switch (obj.type) {
|
||||
case 1: type = "DM"; break;
|
||||
case 3: type = "GROUP"; break;
|
||||
case DISCORD.CHANNEL_TYPE.DM: type = "DM"; break;
|
||||
case DISCORD.CHANNEL_TYPE.GROUP_DM: type = "GROUP"; break;
|
||||
default: return null;
|
||||
}
|
||||
|
||||
@@ -224,7 +248,7 @@ class DISCORD {
|
||||
}
|
||||
};
|
||||
|
||||
if (obj.parent_id) {
|
||||
if (obj.type === DISCORD.CHANNEL_TYPE.ANNOUNCEMENT_THREAD || obj.type === DISCORD.CHANNEL_TYPE.PUBLIC_THREAD || obj.type === DISCORD.CHANNEL_TYPE.PRIVATE_THREAD) {
|
||||
channel["extra"]["parent"] = obj.parent_id;
|
||||
}
|
||||
else {
|
||||
|
@@ -86,12 +86,12 @@ const GUI = (function() {
|
||||
<label><input id='dht-cfg-autoscroll' type='checkbox'> Autoscroll</label><br>
|
||||
<br>
|
||||
<label>After reaching the first message in channel...</label><br>
|
||||
${radio("afm", "nothing", "Do Nothing")}
|
||||
${radio("afm", "nothing", "Continue Tracking")}
|
||||
${radio("afm", "pause", "Pause Tracking")}
|
||||
${radio("afm", "switch", "Switch to Next Channel")}
|
||||
<br>
|
||||
<label>After reaching a previously saved message...</label><br>
|
||||
${radio("asm", "nothing", "Do Nothing")}
|
||||
${radio("asm", "nothing", "Continue Tracking")}
|
||||
${radio("asm", "pause", "Pause Tracking")}
|
||||
${radio("asm", "switch", "Switch to Next Channel")}
|
||||
<p id='dht-cfg-note'>It is recommended to disable link and image previews to avoid putting unnecessary strain on your browser.</p>`;
|
||||
|
@@ -174,12 +174,10 @@ const STATE = (function() {
|
||||
},
|
||||
|
||||
/**
|
||||
* @param {String} channelId
|
||||
* @param {DiscordMessage[]} discordMessageArray
|
||||
*/
|
||||
async addDiscordMessages(channelId, discordMessageArray) {
|
||||
// https://discord.com/developers/docs/resources/channel#message-object-message-types
|
||||
discordMessageArray = discordMessageArray.filter(msg => (msg.type === 0 || msg.type === 19 || msg.type === 21) && msg.state === "SENT");
|
||||
async addDiscordMessages(discordMessageArray) {
|
||||
discordMessageArray = discordMessageArray.filter(msg => (msg.type === DISCORD.MESSAGE_TYPE.DEFAULT || msg.type === DISCORD.MESSAGE_TYPE.REPLY || msg.type === DISCORD.MESSAGE_TYPE.THREAD_STARTER) && msg.state === "SENT");
|
||||
|
||||
if (discordMessageArray.length === 0) {
|
||||
return false;
|
||||
|
@@ -6,6 +6,8 @@
|
||||
|
||||
<script type="text/javascript">
|
||||
window.DHT_EMBEDDED = "/*[ARCHIVE]*/";
|
||||
window.DHT_SERVER_URL = "/*[SERVER_URL]*/";
|
||||
window.DHT_SERVER_TOKEN = "/*[SERVER_TOKEN]*/";
|
||||
/*[JS]*/
|
||||
</script>
|
||||
<style>
|
||||
|
@@ -1,7 +1,8 @@
|
||||
const DISCORD = (function() {
|
||||
const regex = {
|
||||
formatBold: /\*\*([\s\S]+?)\*\*(?!\*)/g,
|
||||
formatItalic: /(.)?([_*])([\s\S]+?)\2(?!\2)/g,
|
||||
formatItalic1: /\*([\s\S]+?)\*(?!\*)/g,
|
||||
formatItalic2: /_([\s\S]+?)_(?!_)\b/g,
|
||||
formatUnderline: /__([\s\S]+?)__(?!_)/g,
|
||||
formatStrike: /~~([\s\S]+?)~~(?!~)/g,
|
||||
formatCodeInline: /(`+)\s*([\s\S]*?[^`])\s*\1(?!`)/g,
|
||||
@@ -48,7 +49,8 @@ const DISCORD = (function() {
|
||||
.replace(regex.specialEscapedDouble, full => full.replace(/\\/g, "").replace(/(.)/g, escapeHtmlMatch))
|
||||
.replace(regex.formatBold, "<b>$1</b>")
|
||||
.replace(regex.formatUnderline, "<u>$1</u>")
|
||||
.replace(regex.formatItalic, (full, pre, char, match) => pre === "\\" ? full : (pre || "") + "<i>" + match + "</i>")
|
||||
.replace(regex.formatItalic1, "<i>$1</i>")
|
||||
.replace(regex.formatItalic2, "<i>$1</i>")
|
||||
.replace(regex.formatStrike, "<s>$1</s>");
|
||||
}
|
||||
|
||||
|
@@ -182,15 +182,32 @@ const STATE = (function() {
|
||||
return null;
|
||||
};
|
||||
|
||||
const getMessageList = function() {
|
||||
const getMessageList = async function(abortSignal) {
|
||||
if (!loadedMessages) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const messages = getMessages(selectedChannel);
|
||||
const startIndex = messagesPerPage * (root.getCurrentPage() - 1);
|
||||
const slicedMessages = loadedMessages.slice(startIndex, !messagesPerPage ? undefined : startIndex + messagesPerPage);
|
||||
|
||||
return loadedMessages.slice(startIndex, !messagesPerPage ? undefined : startIndex + messagesPerPage).map(key => {
|
||||
let messageTexts = null;
|
||||
|
||||
if (window.DHT_SERVER_URL !== null) {
|
||||
const messageIds = new Set(slicedMessages);
|
||||
|
||||
for (const key of slicedMessages) {
|
||||
const message = messages[key];
|
||||
|
||||
if ("r" in message) {
|
||||
messageIds.add(message.r);
|
||||
}
|
||||
}
|
||||
|
||||
messageTexts = await getMessageTextsFromServer(messageIds, abortSignal);
|
||||
}
|
||||
|
||||
return slicedMessages.map(key => {
|
||||
/**
|
||||
* @type {{}}
|
||||
* @property {Number} u
|
||||
@@ -216,6 +233,9 @@ const STATE = (function() {
|
||||
if ("m" in message) {
|
||||
obj["contents"] = message.m;
|
||||
}
|
||||
else if (messageTexts && key in messageTexts) {
|
||||
obj["contents"] = messageTexts[key];
|
||||
}
|
||||
|
||||
if ("e" in message) {
|
||||
obj["embeds"] = message.e.map(embed => JSON.parse(embed));
|
||||
@@ -230,15 +250,16 @@ const STATE = (function() {
|
||||
}
|
||||
|
||||
if ("r" in message) {
|
||||
const replyMessage = getMessageById(message.r);
|
||||
const replyId = message.r;
|
||||
const replyMessage = getMessageById(replyId);
|
||||
const replyUser = replyMessage ? getUser(replyMessage.u) : null;
|
||||
const replyAvatar = replyUser && replyUser.avatar ? { id: getUserId(replyMessage.u), path: replyUser.avatar } : null;
|
||||
|
||||
obj["reply"] = replyMessage ? {
|
||||
"id": message.r,
|
||||
"id": replyId,
|
||||
"user": replyUser,
|
||||
"avatar": replyAvatar,
|
||||
"contents": replyMessage.m
|
||||
"contents": messageTexts != null && replyId in messageTexts ? messageTexts[replyId] : replyMessage.m,
|
||||
} : null;
|
||||
}
|
||||
|
||||
@@ -250,9 +271,35 @@ const STATE = (function() {
|
||||
});
|
||||
};
|
||||
|
||||
const getMessageTextsFromServer = async function(messageIds, abortSignal) {
|
||||
let idParams = "";
|
||||
|
||||
for (const messageId of messageIds) {
|
||||
idParams += "id=" + encodeURIComponent(messageId) + "&";
|
||||
}
|
||||
|
||||
const response = await fetch(DHT_SERVER_URL + "/get-messages?" + idParams + "token=" + encodeURIComponent(DHT_SERVER_TOKEN), {
|
||||
method: "GET",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
credentials: "omit",
|
||||
redirect: "error",
|
||||
signal: abortSignal
|
||||
});
|
||||
|
||||
if (response.status === 200) {
|
||||
return response.json();
|
||||
}
|
||||
else {
|
||||
throw new Error("Server returned status " + response.status + " " + response.statusText);
|
||||
}
|
||||
};
|
||||
|
||||
let eventOnUsersRefreshed;
|
||||
let eventOnChannelsRefreshed;
|
||||
let eventOnMessagesRefreshed;
|
||||
let messageLoaderAborter = null;
|
||||
|
||||
const triggerUsersRefreshed = function() {
|
||||
eventOnUsersRefreshed && eventOnUsersRefreshed(getUserList());
|
||||
@@ -263,7 +310,22 @@ const STATE = (function() {
|
||||
};
|
||||
|
||||
const triggerMessagesRefreshed = function() {
|
||||
eventOnMessagesRefreshed && eventOnMessagesRefreshed(getMessageList());
|
||||
if (!eventOnMessagesRefreshed) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (messageLoaderAborter != null) {
|
||||
messageLoaderAborter.abort();
|
||||
}
|
||||
|
||||
const aborter = new AbortController();
|
||||
messageLoaderAborter = aborter;
|
||||
|
||||
getMessageList(aborter.signal).then(eventOnMessagesRefreshed).finally(() => {
|
||||
if (messageLoaderAborter === aborter) {
|
||||
messageLoaderAborter = null;
|
||||
}
|
||||
});
|
||||
};
|
||||
|
||||
const getFilteredMessageKeys = function(channel) {
|
||||
|
@@ -1,27 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import glob
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
if os.name == "nt":
|
||||
uglifyjs = os.path.abspath("../../lib/uglifyjs.cmd")
|
||||
else:
|
||||
uglifyjs = "uglifyjs"
|
||||
|
||||
if shutil.which(uglifyjs) is None:
|
||||
print("Cannot find executable: {0}".format(uglifyjs))
|
||||
sys.exit(1)
|
||||
|
||||
input_dir = os.path.abspath("./Tracker/scripts")
|
||||
output_dir = os.path.abspath("../Desktop/bin/.res/scripts")
|
||||
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
for file in glob.glob(input_dir + "/*.js"):
|
||||
name = os.path.basename(file)
|
||||
print("Minifying {0}...".format(name))
|
||||
os.system("{0} {1} -o {2}/{3}".format(uglifyjs, file, output_dir, name))
|
||||
|
||||
print("Done!")
|
@@ -4,7 +4,8 @@ public readonly struct Attachment {
|
||||
public ulong Id { get; internal init; }
|
||||
public string Name { get; internal init; }
|
||||
public string? Type { get; internal init; }
|
||||
public string Url { get; internal init; }
|
||||
public string NormalizedUrl { get; internal init; }
|
||||
public string DownloadUrl { get; internal init; }
|
||||
public ulong Size { get; internal init; }
|
||||
public int? Width { get; internal init; }
|
||||
public int? Height { get; internal init; }
|
||||
|
@@ -1,30 +1,33 @@
|
||||
using System;
|
||||
using System.Net;
|
||||
using DHT.Server.Download;
|
||||
|
||||
namespace DHT.Server.Data;
|
||||
|
||||
public readonly struct Download {
|
||||
internal static Download NewSuccess(string url, byte[] data) {
|
||||
return new Download(url, DownloadStatus.Success, (ulong) Math.Max(data.LongLength, 0), data);
|
||||
internal static Download NewSuccess(DownloadItem item, byte[] data) {
|
||||
return new Download(item.NormalizedUrl, item.DownloadUrl, DownloadStatus.Success, (ulong) Math.Max(data.LongLength, 0), data);
|
||||
}
|
||||
|
||||
internal static Download NewFailure(string url, HttpStatusCode? statusCode, ulong size) {
|
||||
return new Download(url, statusCode.HasValue ? (DownloadStatus) (int) statusCode : DownloadStatus.GenericError, size);
|
||||
internal static Download NewFailure(DownloadItem item, HttpStatusCode? statusCode, ulong size) {
|
||||
return new Download(item.NormalizedUrl, item.DownloadUrl, statusCode.HasValue ? (DownloadStatus) (int) statusCode : DownloadStatus.GenericError, size);
|
||||
}
|
||||
|
||||
public string Url { get; }
|
||||
public string NormalizedUrl { get; }
|
||||
public string DownloadUrl { get; }
|
||||
public DownloadStatus Status { get; }
|
||||
public ulong Size { get; }
|
||||
public byte[]? Data { get; }
|
||||
|
||||
internal Download(string url, DownloadStatus status, ulong size, byte[]? data = null) {
|
||||
Url = url;
|
||||
internal Download(string normalizedUrl, string downloadUrl, DownloadStatus status, ulong size, byte[]? data = null) {
|
||||
NormalizedUrl = normalizedUrl;
|
||||
DownloadUrl = downloadUrl;
|
||||
Status = status;
|
||||
Size = size;
|
||||
Data = data;
|
||||
}
|
||||
|
||||
internal Download WithData(byte[] data) {
|
||||
return new Download(Url, Status, Size, data);
|
||||
return new Download(NormalizedUrl, DownloadUrl, Status, Size, data);
|
||||
}
|
||||
}
|
||||
|
@@ -44,7 +44,7 @@ public sealed class DummyDatabaseFile : IDatabaseFile {
|
||||
return 0;
|
||||
}
|
||||
|
||||
public List<Message> GetMessages(MessageFilter? filter = null) {
|
||||
public List<Message> GetMessages(MessageFilter? filter = null, bool includeText = true) {
|
||||
return new();
|
||||
}
|
||||
|
||||
|
3
app/Server/Database/Export/Snowflake.cs
Normal file
3
app/Server/Database/Export/Snowflake.cs
Normal file
@@ -0,0 +1,3 @@
|
||||
namespace DHT.Server.Database.Export;
|
||||
|
||||
readonly record struct Snowflake(ulong Id);
|
23
app/Server/Database/Export/SnowflakeJsonSerializer.cs
Normal file
23
app/Server/Database/Export/SnowflakeJsonSerializer.cs
Normal file
@@ -0,0 +1,23 @@
|
||||
using System;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace DHT.Server.Database.Export;
|
||||
|
||||
sealed class SnowflakeJsonSerializer : JsonConverter<Snowflake> {
|
||||
public override Snowflake Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) {
|
||||
return new Snowflake(ulong.Parse(reader.GetString()!));
|
||||
}
|
||||
|
||||
public override void Write(Utf8JsonWriter writer, Snowflake value, JsonSerializerOptions options) {
|
||||
writer.WriteStringValue(value.Id.ToString());
|
||||
}
|
||||
|
||||
public override Snowflake ReadAsPropertyName(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) {
|
||||
return new Snowflake(ulong.Parse(reader.GetString()!));
|
||||
}
|
||||
|
||||
public override void WriteAsPropertyName(Utf8JsonWriter writer, Snowflake value, JsonSerializerOptions options) {
|
||||
writer.WritePropertyName(value.Id.ToString());
|
||||
}
|
||||
}
|
@@ -3,5 +3,7 @@ using DHT.Server.Data;
|
||||
namespace DHT.Server.Database.Export.Strategy;
|
||||
|
||||
public interface IViewerExportStrategy {
|
||||
bool IncludeMessageText { get; }
|
||||
string ProcessViewerTemplate(string template);
|
||||
string GetAttachmentUrl(Attachment attachment);
|
||||
}
|
||||
|
@@ -12,7 +12,14 @@ public sealed class LiveViewerExportStrategy : IViewerExportStrategy {
|
||||
this.safeToken = WebUtility.UrlEncode(token);
|
||||
}
|
||||
|
||||
public bool IncludeMessageText => false;
|
||||
|
||||
public string ProcessViewerTemplate(string template) {
|
||||
return template.Replace("/*[SERVER_URL]*/", "http://127.0.0.1:" + safePort)
|
||||
.Replace("/*[SERVER_TOKEN]*/", WebUtility.UrlEncode(safeToken));
|
||||
}
|
||||
|
||||
public string GetAttachmentUrl(Attachment attachment) {
|
||||
return "http://127.0.0.1:" + safePort + "/get-attachment/" + WebUtility.UrlEncode(attachment.Url) + "?token=" + safeToken;
|
||||
return "http://127.0.0.1:" + safePort + "/get-attachment/" + WebUtility.UrlEncode(attachment.NormalizedUrl) + "?token=" + safeToken;
|
||||
}
|
||||
}
|
||||
|
@@ -7,7 +7,19 @@ public sealed class StandaloneViewerExportStrategy : IViewerExportStrategy {
|
||||
|
||||
private StandaloneViewerExportStrategy() {}
|
||||
|
||||
public bool IncludeMessageText => true;
|
||||
|
||||
public string ProcessViewerTemplate(string template) {
|
||||
return template.Replace("\"/*[SERVER_URL]*/\"", "null")
|
||||
.Replace("\"/*[SERVER_TOKEN]*/\"", "null");
|
||||
}
|
||||
|
||||
public string GetAttachmentUrl(Attachment attachment) {
|
||||
return attachment.Url;
|
||||
// The normalized URL will not load files from Discord CDN once the time limit is enforced.
|
||||
|
||||
// The downloaded URL would work, but only for a limited time, so it is better for the links to not work
|
||||
// rather than give users a false sense of security.
|
||||
|
||||
return attachment.NormalizedUrl;
|
||||
}
|
||||
}
|
||||
|
93
app/Server/Database/Export/ViewerJson.cs
Normal file
93
app/Server/Database/Export/ViewerJson.cs
Normal file
@@ -0,0 +1,93 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace DHT.Server.Database.Export;
|
||||
|
||||
sealed class ViewerJson {
|
||||
public required JsonMeta Meta { get; init; }
|
||||
public required Dictionary<Snowflake, Dictionary<Snowflake, JsonMessage>> Data { get; init; }
|
||||
|
||||
public sealed class JsonMeta {
|
||||
public required Dictionary<Snowflake, JsonUser> Users { get; init; }
|
||||
public required List<Snowflake> Userindex { get; init; }
|
||||
public required List<JsonServer> Servers { get; init; }
|
||||
public required Dictionary<Snowflake, JsonChannel> Channels { get; init; }
|
||||
}
|
||||
|
||||
public sealed class JsonUser {
|
||||
public required string Name { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? Avatar { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? Tag { get; init; }
|
||||
}
|
||||
|
||||
public sealed class JsonServer {
|
||||
public required string Name { get; init; }
|
||||
public required string Type { get; init; }
|
||||
}
|
||||
|
||||
public sealed class JsonChannel {
|
||||
public required int Server { get; init; }
|
||||
public required string Name { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? Parent { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public int? Position { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? Topic { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public bool? Nsfw { get; init; }
|
||||
}
|
||||
|
||||
public sealed class JsonMessage {
|
||||
public required int U { get; init; }
|
||||
public required long T { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? M { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public long? Te { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? R { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public JsonMessageAttachment[]? A { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string[]? E { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public JsonMessageReaction[]? Re { get; init; }
|
||||
}
|
||||
|
||||
public sealed class JsonMessageAttachment {
|
||||
public required string Url { get; init; }
|
||||
public required string Name { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public int? Width { get; set; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public int? Height { get; set; }
|
||||
}
|
||||
|
||||
public sealed class JsonMessageReaction {
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? Id { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? N { get; init; }
|
||||
|
||||
public required bool A { get; init; }
|
||||
public required int C { get; init; }
|
||||
}
|
||||
}
|
11
app/Server/Database/Export/ViewerJsonContext.cs
Normal file
11
app/Server/Database/Export/ViewerJsonContext.cs
Normal file
@@ -0,0 +1,11 @@
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace DHT.Server.Database.Export;
|
||||
|
||||
[JsonSourceGenerationOptions(
|
||||
Converters = new [] { typeof(SnowflakeJsonSerializer) },
|
||||
PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase,
|
||||
GenerationMode = JsonSourceGenerationMode.Default
|
||||
)]
|
||||
[JsonSerializable(typeof(ViewerJson))]
|
||||
sealed partial class ViewerJsonContext : JsonSerializerContext {}
|
@@ -21,7 +21,7 @@ public static class ViewerJsonExport {
|
||||
var includedChannelIds = new HashSet<ulong>();
|
||||
var includedServerIds = new HashSet<ulong>();
|
||||
|
||||
var includedMessages = db.GetMessages(filter);
|
||||
var includedMessages = db.GetMessages(filter, strategy.IncludeMessageText);
|
||||
var includedChannels = new List<Channel>();
|
||||
|
||||
foreach (var message in includedMessages) {
|
||||
@@ -42,26 +42,28 @@ public static class ViewerJsonExport {
|
||||
|
||||
perf.Step("Collect database data");
|
||||
|
||||
var value = new {
|
||||
meta = new { users, userindex, servers, channels },
|
||||
data = GenerateMessageList(includedMessages, userIndices, strategy),
|
||||
var value = new ViewerJson {
|
||||
Meta = new ViewerJson.JsonMeta {
|
||||
Users = users,
|
||||
Userindex = userindex,
|
||||
Servers = servers,
|
||||
Channels = channels
|
||||
},
|
||||
Data = GenerateMessageList(includedMessages, userIndices, strategy)
|
||||
};
|
||||
|
||||
perf.Step("Generate value object");
|
||||
|
||||
var opts = new JsonSerializerOptions();
|
||||
opts.Converters.Add(new ViewerJsonSnowflakeSerializer());
|
||||
|
||||
await JsonSerializer.SerializeAsync(stream, value, opts);
|
||||
await JsonSerializer.SerializeAsync(stream, value, ViewerJsonContext.Default.ViewerJson);
|
||||
|
||||
perf.Step("Serialize to JSON");
|
||||
perf.End();
|
||||
}
|
||||
|
||||
private static Dictionary<string, object> GenerateUserList(IDatabaseFile db, HashSet<ulong> userIds, out List<string> userindex, out Dictionary<ulong, object> userIndices) {
|
||||
var users = new Dictionary<string, object>();
|
||||
userindex = new List<string>();
|
||||
userIndices = new Dictionary<ulong, object>();
|
||||
private static Dictionary<Snowflake, ViewerJson.JsonUser> GenerateUserList(IDatabaseFile db, HashSet<ulong> userIds, out List<Snowflake> userindex, out Dictionary<ulong, int> userIndices) {
|
||||
var users = new Dictionary<Snowflake, ViewerJson.JsonUser>();
|
||||
userindex = new List<Snowflake>();
|
||||
userIndices = new Dictionary<ulong, int>();
|
||||
|
||||
foreach (var user in db.GetAllUsers()) {
|
||||
var id = user.Id;
|
||||
@@ -69,30 +71,23 @@ public static class ViewerJsonExport {
|
||||
continue;
|
||||
}
|
||||
|
||||
var obj = new Dictionary<string, object> {
|
||||
["name"] = user.Name
|
||||
};
|
||||
|
||||
if (user.AvatarUrl != null) {
|
||||
obj["avatar"] = user.AvatarUrl;
|
||||
}
|
||||
|
||||
if (user.Discriminator != null) {
|
||||
obj["tag"] = user.Discriminator;
|
||||
}
|
||||
|
||||
var idStr = id.ToString();
|
||||
var idSnowflake = new Snowflake(id);
|
||||
userIndices[id] = users.Count;
|
||||
userindex.Add(idStr);
|
||||
users[idStr] = obj;
|
||||
userindex.Add(idSnowflake);
|
||||
|
||||
users[idSnowflake] = new ViewerJson.JsonUser {
|
||||
Name = user.Name,
|
||||
Avatar = user.AvatarUrl,
|
||||
Tag = user.Discriminator
|
||||
};
|
||||
}
|
||||
|
||||
return users;
|
||||
}
|
||||
|
||||
private static List<object> GenerateServerList(IDatabaseFile db, HashSet<ulong> serverIds, out Dictionary<ulong, object> serverIndices) {
|
||||
var servers = new List<object>();
|
||||
serverIndices = new Dictionary<ulong, object>();
|
||||
private static List<ViewerJson.JsonServer> GenerateServerList(IDatabaseFile db, HashSet<ulong> serverIds, out Dictionary<ulong, int> serverIndices) {
|
||||
var servers = new List<ViewerJson.JsonServer>();
|
||||
serverIndices = new Dictionary<ulong, int>();
|
||||
|
||||
foreach (var server in db.GetAllServers()) {
|
||||
var id = server.Id;
|
||||
@@ -101,113 +96,78 @@ public static class ViewerJsonExport {
|
||||
}
|
||||
|
||||
serverIndices[id] = servers.Count;
|
||||
servers.Add(new Dictionary<string, object> {
|
||||
["name"] = server.Name,
|
||||
["type"] = ServerTypes.ToJsonViewerString(server.Type),
|
||||
|
||||
servers.Add(new ViewerJson.JsonServer {
|
||||
Name = server.Name,
|
||||
Type = ServerTypes.ToJsonViewerString(server.Type)
|
||||
});
|
||||
}
|
||||
|
||||
return servers;
|
||||
}
|
||||
|
||||
private static Dictionary<string, object> GenerateChannelList(List<Channel> includedChannels, Dictionary<ulong, object> serverIndices) {
|
||||
var channels = new Dictionary<string, object>();
|
||||
private static Dictionary<Snowflake, ViewerJson.JsonChannel> GenerateChannelList(List<Channel> includedChannels, Dictionary<ulong, int> serverIndices) {
|
||||
var channels = new Dictionary<Snowflake, ViewerJson.JsonChannel>();
|
||||
|
||||
foreach (var channel in includedChannels) {
|
||||
var obj = new Dictionary<string, object> {
|
||||
["server"] = serverIndices[channel.Server],
|
||||
["name"] = channel.Name,
|
||||
var channelIdSnowflake = new Snowflake(channel.Id);
|
||||
|
||||
channels[channelIdSnowflake] = new ViewerJson.JsonChannel {
|
||||
Server = serverIndices[channel.Server],
|
||||
Name = channel.Name,
|
||||
Parent = channel.ParentId?.ToString(),
|
||||
Position = channel.Position,
|
||||
Topic = channel.Topic,
|
||||
Nsfw = channel.Nsfw
|
||||
};
|
||||
|
||||
if (channel.ParentId != null) {
|
||||
obj["parent"] = channel.ParentId;
|
||||
}
|
||||
|
||||
if (channel.Position != null) {
|
||||
obj["position"] = channel.Position;
|
||||
}
|
||||
|
||||
if (channel.Topic != null) {
|
||||
obj["topic"] = channel.Topic;
|
||||
}
|
||||
|
||||
if (channel.Nsfw != null) {
|
||||
obj["nsfw"] = channel.Nsfw;
|
||||
}
|
||||
|
||||
channels[channel.Id.ToString()] = obj;
|
||||
}
|
||||
|
||||
return channels;
|
||||
}
|
||||
|
||||
private static Dictionary<string, Dictionary<string, object>> GenerateMessageList( List<Message> includedMessages, Dictionary<ulong, object> userIndices, IViewerExportStrategy strategy) {
|
||||
var data = new Dictionary<string, Dictionary<string, object>>();
|
||||
private static Dictionary<Snowflake, Dictionary<Snowflake, ViewerJson.JsonMessage>> GenerateMessageList(List<Message> includedMessages, Dictionary<ulong, int> userIndices, IViewerExportStrategy strategy) {
|
||||
var data = new Dictionary<Snowflake, Dictionary<Snowflake, ViewerJson.JsonMessage>>();
|
||||
|
||||
foreach (var grouping in includedMessages.GroupBy(static message => message.Channel)) {
|
||||
var channel = grouping.Key.ToString();
|
||||
var channelData = new Dictionary<string, object>();
|
||||
var channelIdSnowflake = new Snowflake(grouping.Key);
|
||||
var channelData = new Dictionary<Snowflake, ViewerJson.JsonMessage>();
|
||||
|
||||
foreach (var message in grouping) {
|
||||
var obj = new Dictionary<string, object> {
|
||||
["u"] = userIndices[message.Sender],
|
||||
["t"] = message.Timestamp,
|
||||
};
|
||||
|
||||
if (!string.IsNullOrEmpty(message.Text)) {
|
||||
obj["m"] = message.Text;
|
||||
}
|
||||
|
||||
if (message.EditTimestamp != null) {
|
||||
obj["te"] = message.EditTimestamp;
|
||||
}
|
||||
|
||||
if (message.RepliedToId != null) {
|
||||
obj["r"] = message.RepliedToId.Value;
|
||||
}
|
||||
|
||||
if (!message.Attachments.IsEmpty) {
|
||||
obj["a"] = message.Attachments.Select(attachment => {
|
||||
var a = new Dictionary<string, object> {
|
||||
{ "url", strategy.GetAttachmentUrl(attachment) },
|
||||
{ "name", Uri.TryCreate(attachment.Url, UriKind.Absolute, out var uri) ? Path.GetFileName(uri.LocalPath) : attachment.Url },
|
||||
var messageIdSnowflake = new Snowflake(message.Id);
|
||||
|
||||
channelData[messageIdSnowflake] = new ViewerJson.JsonMessage {
|
||||
U = userIndices[message.Sender],
|
||||
T = message.Timestamp,
|
||||
M = string.IsNullOrEmpty(message.Text) ? null : message.Text,
|
||||
Te = message.EditTimestamp,
|
||||
R = message.RepliedToId?.ToString(),
|
||||
|
||||
A = message.Attachments.IsEmpty ? null : message.Attachments.Select(attachment => {
|
||||
var a = new ViewerJson.JsonMessageAttachment {
|
||||
Url = strategy.GetAttachmentUrl(attachment),
|
||||
Name = Uri.TryCreate(attachment.NormalizedUrl, UriKind.Absolute, out var uri) ? Path.GetFileName(uri.LocalPath) : attachment.NormalizedUrl
|
||||
};
|
||||
|
||||
if (attachment is { Width: not null, Height: not null }) {
|
||||
a["width"] = attachment.Width;
|
||||
a["height"] = attachment.Height;
|
||||
a.Width = attachment.Width;
|
||||
a.Height = attachment.Height;
|
||||
}
|
||||
|
||||
return a;
|
||||
}).ToArray();
|
||||
}
|
||||
|
||||
if (!message.Embeds.IsEmpty) {
|
||||
obj["e"] = message.Embeds.Select(static embed => embed.Json).ToArray();
|
||||
}
|
||||
|
||||
if (!message.Reactions.IsEmpty) {
|
||||
obj["re"] = message.Reactions.Select(static reaction => {
|
||||
var r = new Dictionary<string, object>();
|
||||
|
||||
if (reaction.EmojiId != null) {
|
||||
r["id"] = reaction.EmojiId.Value;
|
||||
}
|
||||
|
||||
if (reaction.EmojiName != null) {
|
||||
r["n"] = reaction.EmojiName;
|
||||
}
|
||||
|
||||
r["a"] = reaction.EmojiFlags.HasFlag(EmojiFlags.Animated);
|
||||
r["c"] = reaction.Count;
|
||||
return r;
|
||||
}).ToArray();
|
||||
}
|
||||
|
||||
channelData[message.Id.ToString()] = obj;
|
||||
}).ToArray(),
|
||||
|
||||
E = message.Embeds.IsEmpty ? null : message.Embeds.Select(static embed => embed.Json).ToArray(),
|
||||
|
||||
Re = message.Reactions.IsEmpty ? null : message.Reactions.Select(static reaction => new ViewerJson.JsonMessageReaction {
|
||||
Id = reaction.EmojiId?.ToString(),
|
||||
N = reaction.EmojiName,
|
||||
A = reaction.EmojiFlags.HasFlag(EmojiFlags.Animated),
|
||||
C = reaction.Count
|
||||
}).ToArray()
|
||||
};
|
||||
}
|
||||
|
||||
data[channel] = channelData;
|
||||
data[channelIdSnowflake] = channelData;
|
||||
}
|
||||
|
||||
return data;
|
||||
|
@@ -1,15 +0,0 @@
|
||||
using System;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace DHT.Server.Database.Export;
|
||||
|
||||
sealed class ViewerJsonSnowflakeSerializer : JsonConverter<ulong> {
|
||||
public override ulong Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) {
|
||||
return ulong.Parse(reader.GetString()!);
|
||||
}
|
||||
|
||||
public override void Write(Utf8JsonWriter writer, ulong value, JsonSerializerOptions options) {
|
||||
writer.WriteStringValue(value.ToString());
|
||||
}
|
||||
}
|
@@ -23,7 +23,7 @@ public interface IDatabaseFile : IDisposable {
|
||||
|
||||
void AddMessages(Message[] messages);
|
||||
int CountMessages(MessageFilter? filter = null);
|
||||
List<Message> GetMessages(MessageFilter? filter = null);
|
||||
List<Message> GetMessages(MessageFilter? filter = null, bool includeText = true);
|
||||
HashSet<ulong> GetMessageIds(MessageFilter? filter = null);
|
||||
void RemoveMessages(MessageFilter filter, FilterRemovalMode mode);
|
||||
|
||||
|
23
app/Server/Database/Import/DiscordEmbedLegacyJson.cs
Normal file
23
app/Server/Database/Import/DiscordEmbedLegacyJson.cs
Normal file
@@ -0,0 +1,23 @@
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace DHT.Server.Database.Import;
|
||||
|
||||
sealed class DiscordEmbedLegacyJson {
|
||||
public required string Url { get; init; }
|
||||
public required string Type { get; init; }
|
||||
|
||||
public bool DhtLegacy { get; } = true;
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? Title { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? Description { get; init; }
|
||||
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public ImageJson? Image { get; init; }
|
||||
|
||||
public sealed class ImageJson {
|
||||
public required string Url { get; init; }
|
||||
}
|
||||
}
|
@@ -0,0 +1,7 @@
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace DHT.Server.Database.Import;
|
||||
|
||||
[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.SnakeCaseLower, GenerationMode = JsonSourceGenerationMode.Default)]
|
||||
[JsonSerializable(typeof(DiscordEmbedLegacyJson))]
|
||||
sealed partial class DiscordEmbedLegacyJsonContext : JsonSerializerContext {}
|
@@ -21,7 +21,7 @@ public static class LegacyArchiveImport {
|
||||
|
||||
public static async Task<bool> Read(Stream stream, IDatabaseFile db, FakeSnowflake fakeSnowflake, Func<Data.Server[], Task<Dictionary<Data.Server, ulong>?>> askForServerIds) {
|
||||
var perf = Log.Start();
|
||||
var root = await JsonSerializer.DeserializeAsync<JsonElement>(stream);
|
||||
var root = await JsonSerializer.DeserializeAsync(stream, JsonElementContext.Default.JsonElement);
|
||||
|
||||
try {
|
||||
var meta = root.RequireObject("meta");
|
||||
@@ -197,7 +197,8 @@ public static class LegacyArchiveImport {
|
||||
Id = fakeSnowflake.Next(),
|
||||
Name = name,
|
||||
Type = type,
|
||||
Url = url,
|
||||
NormalizedUrl = url,
|
||||
DownloadUrl = url,
|
||||
Size = 0, // unknown size
|
||||
};
|
||||
}).DistinctByKeyStable(static attachment => {
|
||||
@@ -211,30 +212,17 @@ public static class LegacyArchiveImport {
|
||||
return embedsArray.Where(static embedObj => embedObj.HasKey("url")).Select(embedObj => {
|
||||
string url = embedObj.RequireString("url", path);
|
||||
string type = embedObj.RequireString("type", path);
|
||||
|
||||
var embedJson = new Dictionary<string, object> {
|
||||
{ "url", url },
|
||||
{ "type", type },
|
||||
{ "dht_legacy", true },
|
||||
|
||||
var embed = new DiscordEmbedLegacyJson {
|
||||
Url = url,
|
||||
Type = type,
|
||||
Title = type == "rich" && embedObj.HasKey("t") ? embedObj.RequireString("t", path) : null,
|
||||
Description = type == "rich" && embedObj.HasKey("d") ? embedObj.RequireString("d", path) : null,
|
||||
Image = type == "image" ? new DiscordEmbedLegacyJson.ImageJson { Url = url } : null
|
||||
};
|
||||
|
||||
if (type == "image") {
|
||||
embedJson["image"] = new Dictionary<string, string> {
|
||||
{ "url", url }
|
||||
};
|
||||
}
|
||||
else if (type == "rich") {
|
||||
if (embedObj.HasKey("t")) {
|
||||
embedJson["title"] = embedObj.RequireString("t", path);
|
||||
}
|
||||
|
||||
if (embedObj.HasKey("d")) {
|
||||
embedJson["description"] = embedObj.RequireString("d", path);
|
||||
}
|
||||
}
|
||||
|
||||
return new Embed {
|
||||
Json = JsonSerializer.Serialize(embedJson)
|
||||
Json = JsonSerializer.Serialize(embed, DiscordEmbedLegacyJsonContext.Default.DiscordEmbedLegacyJson)
|
||||
};
|
||||
});
|
||||
}
|
||||
|
15
app/Server/Database/Sqlite/ISchemaUpgradeCallbacks.cs
Normal file
15
app/Server/Database/Sqlite/ISchemaUpgradeCallbacks.cs
Normal file
@@ -0,0 +1,15 @@
|
||||
using System;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace DHT.Server.Database.Sqlite;
|
||||
|
||||
public interface ISchemaUpgradeCallbacks {
|
||||
Task<bool> CanUpgrade();
|
||||
Task Start(int versionSteps, Func<IProgressReporter, Task> doUpgrade);
|
||||
|
||||
public interface IProgressReporter {
|
||||
Task NextVersion();
|
||||
Task MainWork(string message, int finishedItems, int totalItems);
|
||||
Task SubWork(string message, int finishedItems, int totalItems);
|
||||
}
|
||||
}
|
@@ -1,13 +1,15 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Threading.Tasks;
|
||||
using DHT.Server.Database.Exceptions;
|
||||
using DHT.Server.Database.Sqlite.Utils;
|
||||
using DHT.Server.Download;
|
||||
using DHT.Utils.Logging;
|
||||
using Microsoft.Data.Sqlite;
|
||||
|
||||
namespace DHT.Server.Database.Sqlite;
|
||||
|
||||
sealed class Schema {
|
||||
internal const int Version = 5;
|
||||
internal const int Version = 6;
|
||||
|
||||
private static readonly Log Log = Log.ForType<Schema>();
|
||||
|
||||
@@ -17,12 +19,8 @@ sealed class Schema {
|
||||
this.conn = conn;
|
||||
}
|
||||
|
||||
private void Execute(string sql) {
|
||||
conn.Command(sql).ExecuteNonQuery();
|
||||
}
|
||||
|
||||
public async Task<bool> Setup(Func<Task<bool>> checkCanUpgradeSchemas) {
|
||||
Execute(@"CREATE TABLE IF NOT EXISTS metadata (key TEXT PRIMARY KEY, value TEXT)");
|
||||
public async Task<bool> Setup(ISchemaUpgradeCallbacks callbacks) {
|
||||
conn.Execute(@"CREATE TABLE IF NOT EXISTS metadata (key TEXT PRIMARY KEY, value TEXT)");
|
||||
|
||||
var dbVersionStr = conn.SelectScalar("SELECT value FROM metadata WHERE key = 'version'");
|
||||
if (dbVersionStr == null) {
|
||||
@@ -35,137 +33,323 @@ sealed class Schema {
|
||||
throw new DatabaseTooNewException(dbVersion);
|
||||
}
|
||||
else if (dbVersion < Version) {
|
||||
var proceed = await checkCanUpgradeSchemas();
|
||||
var proceed = await callbacks.CanUpgrade();
|
||||
if (!proceed) {
|
||||
return false;
|
||||
}
|
||||
|
||||
UpgradeSchemas(dbVersion);
|
||||
await callbacks.Start(Version - dbVersion, async reporter => await UpgradeSchemas(dbVersion, reporter));
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private void InitializeSchemas() {
|
||||
Execute(@"CREATE TABLE users (
|
||||
id INTEGER PRIMARY KEY NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
avatar_url TEXT,
|
||||
discriminator TEXT)");
|
||||
conn.Execute("""
|
||||
CREATE TABLE users (
|
||||
id INTEGER PRIMARY KEY NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
avatar_url TEXT,
|
||||
discriminator TEXT
|
||||
)
|
||||
""");
|
||||
|
||||
Execute(@"CREATE TABLE servers (
|
||||
id INTEGER PRIMARY KEY NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
type TEXT NOT NULL)");
|
||||
conn.Execute("""
|
||||
CREATE TABLE servers (
|
||||
id INTEGER PRIMARY KEY NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
type TEXT NOT NULL
|
||||
)
|
||||
""");
|
||||
|
||||
Execute(@"CREATE TABLE channels (
|
||||
id INTEGER PRIMARY KEY NOT NULL,
|
||||
server INTEGER NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
parent_id INTEGER,
|
||||
position INTEGER,
|
||||
topic TEXT,
|
||||
nsfw INTEGER)");
|
||||
conn.Execute("""
|
||||
CREATE TABLE channels (
|
||||
id INTEGER PRIMARY KEY NOT NULL,
|
||||
server INTEGER NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
parent_id INTEGER,
|
||||
position INTEGER,
|
||||
topic TEXT,
|
||||
nsfw INTEGER
|
||||
)
|
||||
""");
|
||||
|
||||
Execute(@"CREATE TABLE messages (
|
||||
message_id INTEGER PRIMARY KEY NOT NULL,
|
||||
sender_id INTEGER NOT NULL,
|
||||
channel_id INTEGER NOT NULL,
|
||||
text TEXT NOT NULL,
|
||||
timestamp INTEGER NOT NULL)");
|
||||
conn.Execute("""
|
||||
CREATE TABLE messages (
|
||||
message_id INTEGER PRIMARY KEY NOT NULL,
|
||||
sender_id INTEGER NOT NULL,
|
||||
channel_id INTEGER NOT NULL,
|
||||
text TEXT NOT NULL,
|
||||
timestamp INTEGER NOT NULL
|
||||
)
|
||||
""");
|
||||
|
||||
Execute(@"CREATE TABLE attachments (
|
||||
message_id INTEGER NOT NULL,
|
||||
attachment_id INTEGER NOT NULL PRIMARY KEY NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
type TEXT,
|
||||
url TEXT NOT NULL,
|
||||
size INTEGER NOT NULL,
|
||||
width INTEGER,
|
||||
height INTEGER)");
|
||||
conn.Execute("""
|
||||
CREATE TABLE attachments (
|
||||
message_id INTEGER NOT NULL,
|
||||
attachment_id INTEGER NOT NULL PRIMARY KEY NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
type TEXT,
|
||||
normalized_url TEXT NOT NULL,
|
||||
download_url TEXT,
|
||||
size INTEGER NOT NULL,
|
||||
width INTEGER,
|
||||
height INTEGER
|
||||
)
|
||||
""");
|
||||
|
||||
Execute(@"CREATE TABLE embeds (
|
||||
message_id INTEGER NOT NULL,
|
||||
json TEXT NOT NULL)");
|
||||
conn.Execute("""
|
||||
CREATE TABLE embeds (
|
||||
message_id INTEGER NOT NULL,
|
||||
json TEXT NOT NULL
|
||||
)
|
||||
""");
|
||||
|
||||
Execute(@"CREATE TABLE reactions (
|
||||
message_id INTEGER NOT NULL,
|
||||
emoji_id INTEGER,
|
||||
emoji_name TEXT,
|
||||
emoji_flags INTEGER NOT NULL,
|
||||
count INTEGER NOT NULL)");
|
||||
conn.Execute("""
|
||||
CREATE TABLE downloads (
|
||||
normalized_url TEXT NOT NULL PRIMARY KEY,
|
||||
download_url TEXT,
|
||||
status INTEGER NOT NULL,
|
||||
size INTEGER NOT NULL,
|
||||
blob BLOB
|
||||
)
|
||||
""");
|
||||
|
||||
conn.Execute("""
|
||||
CREATE TABLE reactions (
|
||||
message_id INTEGER NOT NULL,
|
||||
emoji_id INTEGER,
|
||||
emoji_name TEXT,
|
||||
emoji_flags INTEGER NOT NULL,
|
||||
count INTEGER NOT NULL
|
||||
)
|
||||
""");
|
||||
|
||||
CreateMessageEditTimestampTable();
|
||||
CreateMessageRepliedToTable();
|
||||
CreateDownloadsTable();
|
||||
|
||||
Execute("CREATE INDEX attachments_message_ix ON attachments(message_id)");
|
||||
Execute("CREATE INDEX embeds_message_ix ON embeds(message_id)");
|
||||
Execute("CREATE INDEX reactions_message_ix ON reactions(message_id)");
|
||||
conn.Execute("CREATE INDEX attachments_message_ix ON attachments(message_id)");
|
||||
conn.Execute("CREATE INDEX embeds_message_ix ON embeds(message_id)");
|
||||
conn.Execute("CREATE INDEX reactions_message_ix ON reactions(message_id)");
|
||||
|
||||
Execute("INSERT INTO metadata (key, value) VALUES ('version', " + Version + ")");
|
||||
conn.Execute("INSERT INTO metadata (key, value) VALUES ('version', " + Version + ")");
|
||||
}
|
||||
|
||||
private void CreateMessageEditTimestampTable() {
|
||||
Execute(@"CREATE TABLE edit_timestamps (
|
||||
message_id INTEGER PRIMARY KEY NOT NULL,
|
||||
edit_timestamp INTEGER NOT NULL)");
|
||||
conn.Execute("""
|
||||
CREATE TABLE edit_timestamps (
|
||||
message_id INTEGER PRIMARY KEY NOT NULL,
|
||||
edit_timestamp INTEGER NOT NULL
|
||||
)
|
||||
""");
|
||||
}
|
||||
|
||||
private void CreateMessageRepliedToTable() {
|
||||
Execute(@"CREATE TABLE replied_to (
|
||||
message_id INTEGER PRIMARY KEY NOT NULL,
|
||||
replied_to_id INTEGER NOT NULL)");
|
||||
conn.Execute("""
|
||||
CREATE TABLE replied_to (
|
||||
message_id INTEGER PRIMARY KEY NOT NULL,
|
||||
replied_to_id INTEGER NOT NULL
|
||||
)
|
||||
""");
|
||||
}
|
||||
|
||||
private void CreateDownloadsTable() {
|
||||
Execute(@"CREATE TABLE downloads (
|
||||
url TEXT NOT NULL PRIMARY KEY,
|
||||
status INTEGER NOT NULL,
|
||||
size INTEGER NOT NULL,
|
||||
blob BLOB)");
|
||||
private async Task NormalizeAttachmentUrls(ISchemaUpgradeCallbacks.IProgressReporter reporter) {
|
||||
await reporter.SubWork("Preparing attachments...", 0, 0);
|
||||
|
||||
var normalizedUrls = new Dictionary<long, string>();
|
||||
|
||||
await using (var selectCmd = conn.Command("SELECT attachment_id, url FROM attachments")) {
|
||||
await using var reader = await selectCmd.ExecuteReaderAsync();
|
||||
|
||||
while (reader.Read()) {
|
||||
var attachmentId = reader.GetInt64(0);
|
||||
var originalUrl = reader.GetString(1);
|
||||
normalizedUrls[attachmentId] = DiscordCdn.NormalizeUrl(originalUrl);
|
||||
}
|
||||
}
|
||||
|
||||
await using var tx = conn.BeginTransaction();
|
||||
|
||||
int totalUrls = normalizedUrls.Count;
|
||||
int processedUrls = -1;
|
||||
|
||||
await using (var updateCmd = conn.Command("UPDATE attachments SET download_url = url, url = :normalized_url WHERE attachment_id = :attachment_id")) {
|
||||
updateCmd.Parameters.Add(":attachment_id", SqliteType.Integer);
|
||||
updateCmd.Parameters.Add(":normalized_url", SqliteType.Text);
|
||||
|
||||
foreach (var (attachmentId, normalizedUrl) in normalizedUrls) {
|
||||
if (++processedUrls % 1000 == 0) {
|
||||
await reporter.SubWork("Updating URLs...", processedUrls, totalUrls);
|
||||
}
|
||||
|
||||
updateCmd.Set(":attachment_id", attachmentId);
|
||||
updateCmd.Set(":normalized_url", normalizedUrl);
|
||||
updateCmd.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
await reporter.SubWork("Updating URLs...", totalUrls, totalUrls);
|
||||
|
||||
await tx.CommitAsync();
|
||||
}
|
||||
|
||||
private void UpgradeSchemas(int dbVersion) {
|
||||
private async Task NormalizeDownloadUrls(ISchemaUpgradeCallbacks.IProgressReporter reporter) {
|
||||
await reporter.SubWork("Preparing downloads...", 0, 0);
|
||||
|
||||
var normalizedUrlsToOriginalUrls = new Dictionary<string, string>();
|
||||
var duplicateUrlsToDelete = new HashSet<string>();
|
||||
|
||||
await using (var selectCmd = conn.Command("SELECT url FROM downloads ORDER BY CASE WHEN status = 200 THEN 0 ELSE 1 END")) {
|
||||
await using var reader = await selectCmd.ExecuteReaderAsync();
|
||||
|
||||
while (reader.Read()) {
|
||||
var originalUrl = reader.GetString(0);
|
||||
var normalizedUrl = DiscordCdn.NormalizeUrl(originalUrl);
|
||||
|
||||
if (!normalizedUrlsToOriginalUrls.TryAdd(normalizedUrl, originalUrl)) {
|
||||
duplicateUrlsToDelete.Add(originalUrl);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
conn.Execute("PRAGMA cache_size = -20000");
|
||||
|
||||
SqliteTransaction tx;
|
||||
|
||||
await using (tx = conn.BeginTransaction()) {
|
||||
await reporter.SubWork("Deleting duplicates...", 0, 0);
|
||||
|
||||
await using (var deleteCmd = conn.Delete("downloads", ("url", SqliteType.Text))) {
|
||||
foreach (var duplicateUrl in duplicateUrlsToDelete) {
|
||||
deleteCmd.Set(":url", duplicateUrl);
|
||||
deleteCmd.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
await tx.CommitAsync();
|
||||
}
|
||||
|
||||
int totalUrls = normalizedUrlsToOriginalUrls.Count;
|
||||
int processedUrls = -1;
|
||||
|
||||
tx = conn.BeginTransaction();
|
||||
|
||||
await using (var updateCmd = conn.Command("UPDATE downloads SET download_url = :download_url, url = :normalized_url WHERE url = :download_url")) {
|
||||
updateCmd.Parameters.Add(":normalized_url", SqliteType.Text);
|
||||
updateCmd.Parameters.Add(":download_url", SqliteType.Text);
|
||||
|
||||
foreach (var (normalizedUrl, downloadUrl) in normalizedUrlsToOriginalUrls) {
|
||||
if (++processedUrls % 100 == 0) {
|
||||
await reporter.SubWork("Updating URLs...", processedUrls, totalUrls);
|
||||
|
||||
// Not proper way of dealing with transactions, but it avoids a long commit at the end.
|
||||
// Schema upgrades are already non-atomic anyways, so this doesn't make it worse.
|
||||
await tx.CommitAsync();
|
||||
await tx.DisposeAsync();
|
||||
|
||||
tx = conn.BeginTransaction();
|
||||
updateCmd.Transaction = tx;
|
||||
}
|
||||
|
||||
updateCmd.Set(":normalized_url", normalizedUrl);
|
||||
updateCmd.Set(":download_url", downloadUrl);
|
||||
updateCmd.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
await reporter.SubWork("Updating URLs...", totalUrls, totalUrls);
|
||||
|
||||
await tx.CommitAsync();
|
||||
await tx.DisposeAsync();
|
||||
|
||||
conn.Execute("PRAGMA cache_size = -2000");
|
||||
}
|
||||
|
||||
private async Task UpgradeSchemas(int dbVersion, ISchemaUpgradeCallbacks.IProgressReporter reporter) {
|
||||
var perf = Log.Start("from version " + dbVersion);
|
||||
|
||||
Execute("UPDATE metadata SET value = " + Version + " WHERE key = 'version'");
|
||||
conn.Execute("UPDATE metadata SET value = " + Version + " WHERE key = 'version'");
|
||||
|
||||
if (dbVersion <= 1) {
|
||||
Execute("ALTER TABLE channels ADD parent_id INTEGER");
|
||||
await reporter.MainWork("Applying schema changes...", 0, 1);
|
||||
conn.Execute("ALTER TABLE channels ADD parent_id INTEGER");
|
||||
|
||||
perf.Step("Upgrade to version 2");
|
||||
await reporter.NextVersion();
|
||||
}
|
||||
|
||||
if (dbVersion <= 2) {
|
||||
await reporter.MainWork("Applying schema changes...", 0, 1);
|
||||
|
||||
CreateMessageEditTimestampTable();
|
||||
CreateMessageRepliedToTable();
|
||||
|
||||
Execute(@"INSERT INTO edit_timestamps (message_id, edit_timestamp)
|
||||
SELECT message_id, edit_timestamp FROM messages
|
||||
WHERE edit_timestamp IS NOT NULL");
|
||||
conn.Execute("""
|
||||
INSERT INTO edit_timestamps (message_id, edit_timestamp)
|
||||
SELECT message_id, edit_timestamp
|
||||
FROM messages
|
||||
WHERE edit_timestamp IS NOT NULL
|
||||
""");
|
||||
|
||||
Execute(@"INSERT INTO replied_to (message_id, replied_to_id)
|
||||
SELECT message_id, replied_to_id FROM messages
|
||||
WHERE replied_to_id IS NOT NULL");
|
||||
conn.Execute("""
|
||||
INSERT INTO replied_to (message_id, replied_to_id)
|
||||
SELECT message_id, replied_to_id
|
||||
FROM messages
|
||||
WHERE replied_to_id IS NOT NULL
|
||||
""");
|
||||
|
||||
Execute("ALTER TABLE messages DROP COLUMN replied_to_id");
|
||||
Execute("ALTER TABLE messages DROP COLUMN edit_timestamp");
|
||||
conn.Execute("ALTER TABLE messages DROP COLUMN replied_to_id");
|
||||
conn.Execute("ALTER TABLE messages DROP COLUMN edit_timestamp");
|
||||
|
||||
perf.Step("Upgrade to version 3");
|
||||
|
||||
Execute("VACUUM");
|
||||
|
||||
await reporter.MainWork("Vacuuming the database...", 1, 1);
|
||||
conn.Execute("VACUUM");
|
||||
perf.Step("Vacuum");
|
||||
|
||||
await reporter.NextVersion();
|
||||
}
|
||||
|
||||
if (dbVersion <= 3) {
|
||||
CreateDownloadsTable();
|
||||
conn.Execute("""
|
||||
CREATE TABLE downloads (
|
||||
url TEXT NOT NULL PRIMARY KEY,
|
||||
status INTEGER NOT NULL,
|
||||
size INTEGER NOT NULL,
|
||||
blob BLOB
|
||||
)
|
||||
""");
|
||||
|
||||
perf.Step("Upgrade to version 4");
|
||||
await reporter.NextVersion();
|
||||
}
|
||||
|
||||
if (dbVersion <= 4) {
|
||||
Execute("ALTER TABLE attachments ADD width INTEGER");
|
||||
Execute("ALTER TABLE attachments ADD height INTEGER");
|
||||
await reporter.MainWork("Applying schema changes...", 0, 1);
|
||||
conn.Execute("ALTER TABLE attachments ADD width INTEGER");
|
||||
conn.Execute("ALTER TABLE attachments ADD height INTEGER");
|
||||
|
||||
perf.Step("Upgrade to version 5");
|
||||
await reporter.NextVersion();
|
||||
}
|
||||
|
||||
if (dbVersion <= 5) {
|
||||
await reporter.MainWork("Applying schema changes...", 0, 3);
|
||||
conn.Execute("ALTER TABLE attachments ADD download_url TEXT");
|
||||
conn.Execute("ALTER TABLE downloads ADD download_url TEXT");
|
||||
|
||||
await reporter.MainWork("Updating attachments...", 1, 3);
|
||||
await NormalizeAttachmentUrls(reporter);
|
||||
|
||||
await reporter.MainWork("Updating downloads...", 2, 3);
|
||||
await NormalizeDownloadUrls(reporter);
|
||||
|
||||
await reporter.MainWork("Applying schema changes...", 3, 3);
|
||||
conn.Execute("ALTER TABLE attachments RENAME COLUMN url TO normalized_url");
|
||||
conn.Execute("ALTER TABLE downloads RENAME COLUMN url TO normalized_url");
|
||||
|
||||
perf.Step("Upgrade to version 6");
|
||||
await reporter.NextVersion();
|
||||
}
|
||||
|
||||
perf.End();
|
||||
|
@@ -18,7 +18,7 @@ namespace DHT.Server.Database.Sqlite;
|
||||
public sealed class SqliteDatabaseFile : IDatabaseFile {
|
||||
private const int DefaultPoolSize = 5;
|
||||
|
||||
public static async Task<SqliteDatabaseFile?> OpenOrCreate(string path, Func<Task<bool>> checkCanUpgradeSchemas) {
|
||||
public static async Task<SqliteDatabaseFile?> OpenOrCreate(string path, ISchemaUpgradeCallbacks schemaUpgradeCallbacks, TaskScheduler computeTaskResultScheduler) {
|
||||
var connectionString = new SqliteConnectionStringBuilder {
|
||||
DataSource = path,
|
||||
Mode = SqliteOpenMode.ReadWriteCreate,
|
||||
@@ -27,12 +27,16 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
|
||||
var pool = new SqliteConnectionPool(connectionString, DefaultPoolSize);
|
||||
bool wasOpened;
|
||||
|
||||
using (var conn = pool.Take()) {
|
||||
wasOpened = await new Schema(conn).Setup(checkCanUpgradeSchemas);
|
||||
try {
|
||||
using var conn = pool.Take();
|
||||
wasOpened = await new Schema(conn).Setup(schemaUpgradeCallbacks);
|
||||
} catch (Exception) {
|
||||
pool.Dispose();
|
||||
throw;
|
||||
}
|
||||
|
||||
if (wasOpened) {
|
||||
return new SqliteDatabaseFile(path, pool);
|
||||
return new SqliteDatabaseFile(path, pool, computeTaskResultScheduler);
|
||||
}
|
||||
else {
|
||||
pool.Dispose();
|
||||
@@ -49,13 +53,13 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
|
||||
private readonly AsyncValueComputer<long>.Single totalAttachmentsComputer;
|
||||
private readonly AsyncValueComputer<long>.Single totalDownloadsComputer;
|
||||
|
||||
private SqliteDatabaseFile(string path, SqliteConnectionPool pool) {
|
||||
private SqliteDatabaseFile(string path, SqliteConnectionPool pool, TaskScheduler computeTaskResultScheduler) {
|
||||
this.log = Log.ForType(typeof(SqliteDatabaseFile), System.IO.Path.GetFileName(path));
|
||||
this.pool = pool;
|
||||
|
||||
this.totalMessagesComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateMessageStatistics).WithOutdatedResults().BuildWithComputer(ComputeMessageStatistics);
|
||||
this.totalAttachmentsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateAttachmentStatistics).WithOutdatedResults().BuildWithComputer(ComputeAttachmentStatistics);
|
||||
this.totalDownloadsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateDownloadStatistics).WithOutdatedResults().BuildWithComputer(ComputeDownloadStatistics);
|
||||
this.totalMessagesComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateMessageStatistics, computeTaskResultScheduler).WithOutdatedResults().BuildWithComputer(ComputeMessageStatistics);
|
||||
this.totalAttachmentsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateAttachmentStatistics, computeTaskResultScheduler).WithOutdatedResults().BuildWithComputer(ComputeAttachmentStatistics);
|
||||
this.totalDownloadsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateDownloadStatistics, computeTaskResultScheduler).WithOutdatedResults().BuildWithComputer(ComputeDownloadStatistics);
|
||||
|
||||
this.Path = path;
|
||||
this.Statistics = new DatabaseStatistics();
|
||||
@@ -252,7 +256,8 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
|
||||
("attachment_id", SqliteType.Integer),
|
||||
("name", SqliteType.Text),
|
||||
("type", SqliteType.Text),
|
||||
("url", SqliteType.Text),
|
||||
("normalized_url", SqliteType.Text),
|
||||
("download_url", SqliteType.Text),
|
||||
("size", SqliteType.Integer),
|
||||
("width", SqliteType.Integer),
|
||||
("height", SqliteType.Integer),
|
||||
@@ -308,7 +313,8 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
|
||||
attachmentCmd.Set(":attachment_id", attachment.Id);
|
||||
attachmentCmd.Set(":name", attachment.Name);
|
||||
attachmentCmd.Set(":type", attachment.Type);
|
||||
attachmentCmd.Set(":url", attachment.Url);
|
||||
attachmentCmd.Set(":normalized_url", attachment.NormalizedUrl);
|
||||
attachmentCmd.Set(":download_url", attachment.DownloadUrl);
|
||||
attachmentCmd.Set(":size", attachment.Size);
|
||||
attachmentCmd.Set(":width", attachment.Width);
|
||||
attachmentCmd.Set(":height", attachment.Height);
|
||||
@@ -354,7 +360,7 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
|
||||
return reader.Read() ? reader.GetInt32(0) : 0;
|
||||
}
|
||||
|
||||
public List<Message> GetMessages(MessageFilter? filter = null) {
|
||||
public List<Message> GetMessages(MessageFilter? filter = null, bool includeText = true) {
|
||||
var perf = log.Start();
|
||||
var list = new List<Message>();
|
||||
|
||||
@@ -363,11 +369,13 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
|
||||
var reactions = GetAllReactions();
|
||||
|
||||
using var conn = pool.Take();
|
||||
using var cmd = conn.Command(@"
|
||||
SELECT m.message_id, m.sender_id, m.channel_id, m.text, m.timestamp, et.edit_timestamp, rt.replied_to_id
|
||||
FROM messages m
|
||||
LEFT JOIN edit_timestamps et ON m.message_id = et.message_id
|
||||
LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereClause("m"));
|
||||
using var cmd = conn.Command($"""
|
||||
SELECT m.message_id, m.sender_id, m.channel_id, {(includeText ? "m.text" : "NULL")}, m.timestamp, et.edit_timestamp, rt.replied_to_id
|
||||
FROM messages m
|
||||
LEFT JOIN edit_timestamps et ON m.message_id = et.message_id
|
||||
LEFT JOIN replied_to rt ON m.message_id = rt.message_id
|
||||
{filter.GenerateWhereClause("m")}
|
||||
""");
|
||||
using var reader = cmd.ExecuteReader();
|
||||
|
||||
while (reader.Read()) {
|
||||
@@ -377,7 +385,7 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
|
||||
Id = id,
|
||||
Sender = reader.GetUint64(1),
|
||||
Channel = reader.GetUint64(2),
|
||||
Text = reader.GetString(3),
|
||||
Text = includeText ? reader.GetString(3) : string.Empty,
|
||||
Timestamp = reader.GetInt64(4),
|
||||
EditTimestamp = reader.IsDBNull(5) ? null : reader.GetInt64(5),
|
||||
RepliedToId = reader.IsDBNull(6) ? null : reader.GetUint64(6),
|
||||
@@ -418,7 +426,7 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
|
||||
|
||||
public int CountAttachments(AttachmentFilter? filter = null) {
|
||||
using var conn = pool.Take();
|
||||
using var cmd = conn.Command("SELECT COUNT(DISTINCT url) FROM attachments a" + filter.GenerateWhereClause("a"));
|
||||
using var cmd = conn.Command("SELECT COUNT(DISTINCT normalized_url) FROM attachments a" + filter.GenerateWhereClause("a"));
|
||||
using var reader = cmd.ExecuteReader();
|
||||
|
||||
return reader.Read() ? reader.GetInt32(0) : 0;
|
||||
@@ -427,13 +435,15 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
|
||||
public void AddDownload(Data.Download download) {
|
||||
using var conn = pool.Take();
|
||||
using var cmd = conn.Upsert("downloads", new[] {
|
||||
("url", SqliteType.Text),
|
||||
("normalized_url", SqliteType.Text),
|
||||
("download_url", SqliteType.Text),
|
||||
("status", SqliteType.Integer),
|
||||
("size", SqliteType.Integer),
|
||||
("blob", SqliteType.Blob),
|
||||
});
|
||||
|
||||
cmd.Set(":url", download.Url);
|
||||
cmd.Set(":normalized_url", download.NormalizedUrl);
|
||||
cmd.Set(":download_url", download.DownloadUrl);
|
||||
cmd.Set(":status", (int) download.Status);
|
||||
cmd.Set(":size", download.Size);
|
||||
cmd.Set(":blob", download.Data);
|
||||
@@ -446,15 +456,16 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
|
||||
var list = new List<Data.Download>();
|
||||
|
||||
using var conn = pool.Take();
|
||||
using var cmd = conn.Command("SELECT url, status, size FROM downloads");
|
||||
using var cmd = conn.Command("SELECT normalized_url, download_url, status, size FROM downloads");
|
||||
using var reader = cmd.ExecuteReader();
|
||||
|
||||
while (reader.Read()) {
|
||||
string url = reader.GetString(0);
|
||||
var status = (DownloadStatus) reader.GetInt32(1);
|
||||
ulong size = reader.GetUint64(2);
|
||||
string normalizedUrl = reader.GetString(0);
|
||||
string downloadUrl = reader.GetString(1);
|
||||
var status = (DownloadStatus) reader.GetInt32(2);
|
||||
ulong size = reader.GetUint64(3);
|
||||
|
||||
list.Add(new Data.Download(url, status, size));
|
||||
list.Add(new Data.Download(normalizedUrl, downloadUrl, status, size));
|
||||
}
|
||||
|
||||
return list;
|
||||
@@ -462,8 +473,8 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
|
||||
|
||||
public Data.Download GetDownloadWithData(Data.Download download) {
|
||||
using var conn = pool.Take();
|
||||
using var cmd = conn.Command("SELECT blob FROM downloads WHERE url = :url");
|
||||
cmd.AddAndSet(":url", SqliteType.Text, download.Url);
|
||||
using var cmd = conn.Command("SELECT blob FROM downloads WHERE normalized_url = :url");
|
||||
cmd.AddAndSet(":url", SqliteType.Text, download.NormalizedUrl);
|
||||
|
||||
using var reader = cmd.ExecuteReader();
|
||||
|
||||
@@ -475,14 +486,15 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
|
||||
}
|
||||
}
|
||||
|
||||
public DownloadedAttachment? GetDownloadedAttachment(string url) {
|
||||
public DownloadedAttachment? GetDownloadedAttachment(string normalizedUrl) {
|
||||
using var conn = pool.Take();
|
||||
using var cmd = conn.Command(@"
|
||||
SELECT a.type, d.blob FROM downloads d
|
||||
LEFT JOIN attachments a ON d.url = a.url
|
||||
WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
|
||||
using var cmd = conn.Command("""
|
||||
SELECT a.type, d.blob FROM downloads d
|
||||
LEFT JOIN attachments a ON d.normalized_url = a.normalized_url
|
||||
WHERE d.normalized_url = :normalized_url AND d.status = :success AND d.blob IS NOT NULL
|
||||
""");
|
||||
|
||||
cmd.AddAndSet(":url", SqliteType.Text, url);
|
||||
cmd.AddAndSet(":normalized_url", SqliteType.Text, normalizedUrl);
|
||||
cmd.AddAndSet(":success", SqliteType.Integer, (int) DownloadStatus.Success);
|
||||
|
||||
using var reader = cmd.ExecuteReader();
|
||||
@@ -499,7 +511,13 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
|
||||
|
||||
public void EnqueueDownloadItems(AttachmentFilter? filter = null) {
|
||||
using var conn = pool.Take();
|
||||
using var cmd = conn.Command("INSERT INTO downloads (url, status, size) SELECT a.url, :enqueued, MAX(a.size) FROM attachments a" + filter.GenerateWhereClause("a") + " GROUP BY a.url");
|
||||
using var cmd = conn.Command($"""
|
||||
INSERT INTO downloads (normalized_url, download_url, status, size)
|
||||
SELECT a.normalized_url, a.download_url, :enqueued, MAX(a.size)
|
||||
FROM attachments a
|
||||
{filter.GenerateWhereClause("a")}
|
||||
GROUP BY a.normalized_url
|
||||
""");
|
||||
cmd.AddAndSet(":enqueued", SqliteType.Integer, (int) DownloadStatus.Enqueued);
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
@@ -508,7 +526,7 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
|
||||
var list = new List<DownloadItem>();
|
||||
|
||||
using var conn = pool.Take();
|
||||
using var cmd = conn.Command("SELECT url, size FROM downloads WHERE status = :enqueued LIMIT :limit");
|
||||
using var cmd = conn.Command("SELECT normalized_url, download_url, size FROM downloads WHERE status = :enqueued LIMIT :limit");
|
||||
cmd.AddAndSet(":enqueued", SqliteType.Integer, (int) DownloadStatus.Enqueued);
|
||||
cmd.AddAndSet(":limit", SqliteType.Integer, Math.Max(0, count));
|
||||
|
||||
@@ -516,8 +534,9 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
|
||||
|
||||
while (reader.Read()) {
|
||||
list.Add(new DownloadItem {
|
||||
Url = reader.GetString(0),
|
||||
Size = reader.GetUint64(1),
|
||||
NormalizedUrl = reader.GetString(0),
|
||||
DownloadUrl = reader.GetString(1),
|
||||
Size = reader.GetUint64(2),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -531,7 +550,7 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
|
||||
|
||||
public DownloadStatusStatistics GetDownloadStatusStatistics() {
|
||||
static void LoadUndownloadedStatistics(ISqliteConnection conn, DownloadStatusStatistics result) {
|
||||
using var cmd = conn.Command("SELECT IFNULL(COUNT(size), 0), IFNULL(SUM(size), 0) FROM (SELECT MAX(a.size) size FROM attachments a WHERE a.url NOT IN (SELECT d.url FROM downloads d) GROUP BY a.url)");
|
||||
using var cmd = conn.Command("SELECT IFNULL(COUNT(size), 0), IFNULL(SUM(size), 0) FROM (SELECT MAX(a.size) size FROM attachments a WHERE a.normalized_url NOT IN (SELECT d.normalized_url FROM downloads d) GROUP BY a.normalized_url)");
|
||||
using var reader = cmd.ExecuteReader();
|
||||
|
||||
if (reader.Read()) {
|
||||
@@ -541,14 +560,16 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
|
||||
}
|
||||
|
||||
static void LoadSuccessStatistics(ISqliteConnection conn, DownloadStatusStatistics result) {
|
||||
using var cmd = conn.Command(@"SELECT
|
||||
IFNULL(SUM(CASE WHEN status = :enqueued THEN 1 ELSE 0 END), 0),
|
||||
IFNULL(SUM(CASE WHEN status = :enqueued THEN size ELSE 0 END), 0),
|
||||
IFNULL(SUM(CASE WHEN status = :success THEN 1 ELSE 0 END), 0),
|
||||
IFNULL(SUM(CASE WHEN status = :success THEN size ELSE 0 END), 0),
|
||||
IFNULL(SUM(CASE WHEN status != :enqueued AND status != :success THEN 1 ELSE 0 END), 0),
|
||||
IFNULL(SUM(CASE WHEN status != :enqueued AND status != :success THEN size ELSE 0 END), 0)
|
||||
FROM downloads");
|
||||
using var cmd = conn.Command("""
|
||||
SELECT
|
||||
IFNULL(SUM(CASE WHEN status = :enqueued THEN 1 ELSE 0 END), 0),
|
||||
IFNULL(SUM(CASE WHEN status = :enqueued THEN size ELSE 0 END), 0),
|
||||
IFNULL(SUM(CASE WHEN status = :success THEN 1 ELSE 0 END), 0),
|
||||
IFNULL(SUM(CASE WHEN status = :success THEN size ELSE 0 END), 0),
|
||||
IFNULL(SUM(CASE WHEN status != :enqueued AND status != :success THEN 1 ELSE 0 END), 0),
|
||||
IFNULL(SUM(CASE WHEN status != :enqueued AND status != :success THEN size ELSE 0 END), 0)
|
||||
FROM downloads
|
||||
""");
|
||||
cmd.AddAndSet(":enqueued", SqliteType.Integer, (int) DownloadStatus.Enqueued);
|
||||
cmd.AddAndSet(":success", SqliteType.Integer, (int) DownloadStatus.Success);
|
||||
|
||||
@@ -576,7 +597,7 @@ FROM downloads");
|
||||
var dict = new MultiDictionary<ulong, Attachment>();
|
||||
|
||||
using var conn = pool.Take();
|
||||
using var cmd = conn.Command("SELECT message_id, attachment_id, name, type, url, size, width, height FROM attachments");
|
||||
using var cmd = conn.Command("SELECT message_id, attachment_id, name, type, normalized_url, download_url, size, width, height FROM attachments");
|
||||
using var reader = cmd.ExecuteReader();
|
||||
|
||||
while (reader.Read()) {
|
||||
@@ -586,10 +607,11 @@ FROM downloads");
|
||||
Id = reader.GetUint64(1),
|
||||
Name = reader.GetString(2),
|
||||
Type = reader.IsDBNull(3) ? null : reader.GetString(3),
|
||||
Url = reader.GetString(4),
|
||||
Size = reader.GetUint64(5),
|
||||
Width = reader.IsDBNull(6) ? null : reader.GetInt32(6),
|
||||
Height = reader.IsDBNull(7) ? null : reader.GetInt32(7),
|
||||
NormalizedUrl = reader.GetString(4),
|
||||
DownloadUrl = reader.GetString(5),
|
||||
Size = reader.GetUint64(6),
|
||||
Width = reader.IsDBNull(7) ? null : reader.GetInt32(7),
|
||||
Height = reader.IsDBNull(8) ? null : reader.GetInt32(8),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -677,7 +699,7 @@ FROM downloads");
|
||||
|
||||
private long ComputeAttachmentStatistics() {
|
||||
using var conn = pool.Take();
|
||||
return conn.SelectScalar("SELECT COUNT(DISTINCT url) FROM attachments") as long? ?? 0L;
|
||||
return conn.SelectScalar("SELECT COUNT(DISTINCT normalized_url) FROM attachments") as long? ?? 0L;
|
||||
}
|
||||
|
||||
private void UpdateAttachmentStatistics(long totalAttachments) {
|
||||
|
@@ -50,10 +50,10 @@ static class SqliteFilters {
|
||||
}
|
||||
|
||||
if (filter.DownloadItemRule == AttachmentFilter.DownloadItemRules.OnlyNotPresent) {
|
||||
where.AddCondition("url NOT IN (SELECT url FROM downloads)");
|
||||
where.AddCondition("normalized_url NOT IN (SELECT normalized_url FROM downloads)");
|
||||
}
|
||||
else if (filter.DownloadItemRule == AttachmentFilter.DownloadItemRules.OnlyPresent) {
|
||||
where.AddCondition("url IN (SELECT url FROM downloads)");
|
||||
where.AddCondition("normalized_url IN (SELECT normalized_url FROM downloads)");
|
||||
}
|
||||
|
||||
return where.Generate();
|
||||
|
@@ -5,14 +5,19 @@ using Microsoft.Data.Sqlite;
|
||||
namespace DHT.Server.Database.Sqlite.Utils;
|
||||
|
||||
static class SqliteExtensions {
|
||||
public static SqliteTransaction BeginTransaction(this ISqliteConnection conn) {
|
||||
return conn.InnerConnection.BeginTransaction();
|
||||
}
|
||||
|
||||
public static SqliteCommand Command(this ISqliteConnection conn, string sql) {
|
||||
var cmd = conn.InnerConnection.CreateCommand();
|
||||
cmd.CommandText = sql;
|
||||
return cmd;
|
||||
}
|
||||
|
||||
public static SqliteTransaction BeginTransaction(this ISqliteConnection conn) {
|
||||
return conn.InnerConnection.BeginTransaction();
|
||||
public static void Execute(this ISqliteConnection conn, string sql) {
|
||||
using var cmd = conn.Command(sql);
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
public static object? SelectScalar(this ISqliteConnection conn, string sql) {
|
||||
|
@@ -87,16 +87,16 @@ public sealed class BackgroundDownloadThread : BaseModel {
|
||||
FillQueue(db, queue, cancellationToken);
|
||||
|
||||
while (!cancellationToken.IsCancellationRequested && queue.TryDequeue(out var item)) {
|
||||
var url = item.Url;
|
||||
Log.Debug("Downloading " + url + "...");
|
||||
var downloadUrl = item.DownloadUrl;
|
||||
Log.Debug("Downloading " + downloadUrl + "...");
|
||||
|
||||
try {
|
||||
db.AddDownload(Data.Download.NewSuccess(url, await client.GetByteArrayAsync(url, cancellationToken)));
|
||||
db.AddDownload(Data.Download.NewSuccess(item, await client.GetByteArrayAsync(downloadUrl, cancellationToken)));
|
||||
} catch (HttpRequestException e) {
|
||||
db.AddDownload(Data.Download.NewFailure(url, e.StatusCode, item.Size));
|
||||
db.AddDownload(Data.Download.NewFailure(item, e.StatusCode, item.Size));
|
||||
Log.Error(e);
|
||||
} catch (Exception e) {
|
||||
db.AddDownload(Data.Download.NewFailure(url, null, item.Size));
|
||||
db.AddDownload(Data.Download.NewFailure(item, null, item.Size));
|
||||
Log.Error(e);
|
||||
} finally {
|
||||
parameters.FireOnItemFinished(item);
|
||||
|
15
app/Server/Download/DiscordCdn.cs
Normal file
15
app/Server/Download/DiscordCdn.cs
Normal file
@@ -0,0 +1,15 @@
|
||||
using System;
|
||||
using System.Collections.Frozen;
|
||||
|
||||
namespace DHT.Server.Download;
|
||||
|
||||
static class DiscordCdn {
|
||||
private static FrozenSet<string> CdnHosts { get; } = new [] {
|
||||
"cdn.discordapp.com",
|
||||
"cdn.discord.com",
|
||||
}.ToFrozenSet();
|
||||
|
||||
public static string NormalizeUrl(string originalUrl) {
|
||||
return Uri.TryCreate(originalUrl, UriKind.Absolute, out var uri) && CdnHosts.Contains(uri.Host) ? uri.GetLeftPart(UriPartial.Path) : originalUrl;
|
||||
}
|
||||
}
|
@@ -1,6 +1,7 @@
|
||||
namespace DHT.Server.Download;
|
||||
|
||||
public readonly struct DownloadItem {
|
||||
public string Url { get; init; }
|
||||
public string NormalizedUrl { get; init; }
|
||||
public string DownloadUrl { get; init; }
|
||||
public ulong Size { get; init; }
|
||||
}
|
||||
|
@@ -3,12 +3,9 @@ using System.Net;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using DHT.Server.Database;
|
||||
using DHT.Server.Service;
|
||||
using DHT.Utils.Http;
|
||||
using DHT.Utils.Logging;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Http.Extensions;
|
||||
using Microsoft.Extensions.Primitives;
|
||||
|
||||
namespace DHT.Server.Endpoints;
|
||||
|
||||
@@ -16,25 +13,14 @@ abstract class BaseEndpoint {
|
||||
private static readonly Log Log = Log.ForType<BaseEndpoint>();
|
||||
|
||||
protected IDatabaseFile Db { get; }
|
||||
private readonly ServerParameters parameters;
|
||||
|
||||
protected BaseEndpoint(IDatabaseFile db, ServerParameters parameters) {
|
||||
protected BaseEndpoint(IDatabaseFile db) {
|
||||
this.Db = db;
|
||||
this.parameters = parameters;
|
||||
}
|
||||
|
||||
private async Task Handle(HttpContext ctx, StringValues token) {
|
||||
var request = ctx.Request;
|
||||
public async Task Handle(HttpContext ctx) {
|
||||
var response = ctx.Response;
|
||||
|
||||
Log.Info("Request: " + request.GetDisplayUrl() + " (" + request.ContentLength + " B)");
|
||||
|
||||
if (token.Count != 1 || token[0] != parameters.Token) {
|
||||
Log.Error("Token: " + (token.Count == 1 ? token[0] : "<missing>"));
|
||||
response.StatusCode = (int) HttpStatusCode.Forbidden;
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
response.StatusCode = (int) HttpStatusCode.OK;
|
||||
var output = await Respond(ctx);
|
||||
@@ -49,17 +35,13 @@ abstract class BaseEndpoint {
|
||||
}
|
||||
}
|
||||
|
||||
public async Task HandleGet(HttpContext ctx) {
|
||||
await Handle(ctx, ctx.Request.Query["token"]);
|
||||
}
|
||||
|
||||
public async Task HandlePost(HttpContext ctx) {
|
||||
await Handle(ctx, ctx.Request.Headers["X-DHT-Token"]);
|
||||
}
|
||||
|
||||
protected abstract Task<IHttpOutput> Respond(HttpContext ctx);
|
||||
|
||||
protected static async Task<JsonElement> ReadJson(HttpContext ctx) {
|
||||
return await ctx.Request.ReadFromJsonAsync<JsonElement?>() ?? throw new HttpException(HttpStatusCode.UnsupportedMediaType, "This endpoint only accepts JSON.");
|
||||
try {
|
||||
return await ctx.Request.ReadFromJsonAsync(JsonElementContext.Default.JsonElement);
|
||||
} catch (JsonException) {
|
||||
throw new HttpException(HttpStatusCode.UnsupportedMediaType, "This endpoint only accepts JSON.");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
@@ -2,14 +2,13 @@ using System.Net;
|
||||
using System.Threading.Tasks;
|
||||
using DHT.Server.Data;
|
||||
using DHT.Server.Database;
|
||||
using DHT.Server.Service;
|
||||
using DHT.Utils.Http;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
|
||||
namespace DHT.Server.Endpoints;
|
||||
|
||||
sealed class GetAttachmentEndpoint : BaseEndpoint {
|
||||
public GetAttachmentEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db, parameters) {}
|
||||
public GetAttachmentEndpoint(IDatabaseFile db) : base(db) {}
|
||||
|
||||
protected override Task<IHttpOutput> Respond(HttpContext ctx) {
|
||||
string attachmentUrl = WebUtility.UrlDecode((string) ctx.Request.RouteValues["url"]!);
|
||||
|
34
app/Server/Endpoints/GetMessagesEndpoint.cs
Normal file
34
app/Server/Endpoints/GetMessagesEndpoint.cs
Normal file
@@ -0,0 +1,34 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Net;
|
||||
using System.Threading.Tasks;
|
||||
using DHT.Server.Data.Filters;
|
||||
using DHT.Server.Database;
|
||||
using DHT.Utils.Http;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using GetMessagesJsonContext = DHT.Server.Endpoints.Responses.GetMessagesJsonContext;
|
||||
|
||||
namespace DHT.Server.Endpoints;
|
||||
|
||||
sealed class GetMessagesEndpoint : BaseEndpoint {
|
||||
public GetMessagesEndpoint(IDatabaseFile db) : base(db) {}
|
||||
|
||||
protected override Task<IHttpOutput> Respond(HttpContext ctx) {
|
||||
HashSet<ulong> messageIdSet;
|
||||
try {
|
||||
var messageIds = ctx.Request.Query["id"];
|
||||
messageIdSet = messageIds.Select(ulong.Parse!).ToHashSet();
|
||||
} catch (Exception) {
|
||||
throw new HttpException(HttpStatusCode.BadRequest, "Invalid message ids.");
|
||||
}
|
||||
|
||||
var messageFilter = new MessageFilter {
|
||||
MessageIds = messageIdSet
|
||||
};
|
||||
|
||||
var messages = Db.GetMessages(messageFilter).ToDictionary(static message => message.Id, static message => message.Text);
|
||||
var response = new HttpOutput.Json<Dictionary<ulong, string>>(messages, GetMessagesJsonContext.Default.DictionaryUInt64String);
|
||||
return Task.FromResult<IHttpOutput>(response);
|
||||
}
|
||||
}
|
34
app/Server/Endpoints/GetTrackingScriptEndpoint.cs
Normal file
34
app/Server/Endpoints/GetTrackingScriptEndpoint.cs
Normal file
@@ -0,0 +1,34 @@
|
||||
using System.Reflection;
|
||||
using System.Text;
|
||||
using System.Threading.Tasks;
|
||||
using System.Web;
|
||||
using DHT.Server.Database;
|
||||
using DHT.Server.Service;
|
||||
using DHT.Utils.Http;
|
||||
using DHT.Utils.Resources;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
|
||||
namespace DHT.Server.Endpoints;
|
||||
|
||||
sealed class GetTrackingScriptEndpoint : BaseEndpoint {
|
||||
private static ResourceLoader Resources { get; } = new (Assembly.GetExecutingAssembly());
|
||||
|
||||
private readonly ServerParameters serverParameters;
|
||||
|
||||
public GetTrackingScriptEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db) {
|
||||
serverParameters = parameters;
|
||||
}
|
||||
|
||||
protected override async Task<IHttpOutput> Respond(HttpContext ctx) {
|
||||
string bootstrap = await Resources.ReadTextAsync("Tracker/bootstrap.js");
|
||||
string script = bootstrap.Replace("= 0; /*[PORT]*/", "= " + serverParameters.Port + ";")
|
||||
.Replace("/*[TOKEN]*/", HttpUtility.JavaScriptStringEncode(serverParameters.Token))
|
||||
.Replace("/*[IMPORTS]*/", await Resources.ReadJoinedAsync("Tracker/scripts/", '\n'))
|
||||
.Replace("/*[CSS-CONTROLLER]*/", await Resources.ReadTextAsync("Tracker/styles/controller.css"))
|
||||
.Replace("/*[CSS-SETTINGS]*/", await Resources.ReadTextAsync("Tracker/styles/settings.css"))
|
||||
.Replace("/*[DEBUGGER]*/", ctx.Request.Query.ContainsKey("debug") ? "debugger;" : "");
|
||||
|
||||
ctx.Response.Headers.Append("X-DHT", "1");
|
||||
return new HttpOutput.File("text/javascript", Encoding.UTF8.GetBytes(script));
|
||||
}
|
||||
}
|
8
app/Server/Endpoints/Responses/GetMessagesJsonContext.cs
Normal file
8
app/Server/Endpoints/Responses/GetMessagesJsonContext.cs
Normal file
@@ -0,0 +1,8 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace DHT.Server.Endpoints.Responses;
|
||||
|
||||
[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase, GenerationMode = JsonSourceGenerationMode.Default)]
|
||||
[JsonSerializable(typeof(Dictionary<ulong, string>))]
|
||||
sealed partial class GetMessagesJsonContext : JsonSerializerContext {}
|
@@ -3,14 +3,13 @@ using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using DHT.Server.Data;
|
||||
using DHT.Server.Database;
|
||||
using DHT.Server.Service;
|
||||
using DHT.Utils.Http;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
|
||||
namespace DHT.Server.Endpoints;
|
||||
|
||||
sealed class TrackChannelEndpoint : BaseEndpoint {
|
||||
public TrackChannelEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db, parameters) {}
|
||||
public TrackChannelEndpoint(IDatabaseFile db) : base(db) {}
|
||||
|
||||
protected override async Task<IHttpOutput> Respond(HttpContext ctx) {
|
||||
var root = await ReadJson(ctx);
|
||||
|
@@ -8,7 +8,7 @@ using System.Threading.Tasks;
|
||||
using DHT.Server.Data;
|
||||
using DHT.Server.Data.Filters;
|
||||
using DHT.Server.Database;
|
||||
using DHT.Server.Service;
|
||||
using DHT.Server.Download;
|
||||
using DHT.Utils.Collections;
|
||||
using DHT.Utils.Http;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
@@ -16,7 +16,10 @@ using Microsoft.AspNetCore.Http;
|
||||
namespace DHT.Server.Endpoints;
|
||||
|
||||
sealed class TrackMessagesEndpoint : BaseEndpoint {
|
||||
public TrackMessagesEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db, parameters) {}
|
||||
private const string HasNewMessages = "1";
|
||||
private const string NoNewMessages = "0";
|
||||
|
||||
public TrackMessagesEndpoint(IDatabaseFile db) : base(db) {}
|
||||
|
||||
protected override async Task<IHttpOutput> Respond(HttpContext ctx) {
|
||||
var root = await ReadJson(ctx);
|
||||
@@ -40,7 +43,7 @@ sealed class TrackMessagesEndpoint : BaseEndpoint {
|
||||
|
||||
Db.AddMessages(messages);
|
||||
|
||||
return new HttpOutput.Json(anyNewMessages ? 1 : 0);
|
||||
return new HttpOutput.Text(anyNewMessages ? HasNewMessages : NoNewMessages);
|
||||
}
|
||||
|
||||
private static Message ReadMessage(JsonElement json, string path) => new() {
|
||||
@@ -57,14 +60,18 @@ sealed class TrackMessagesEndpoint : BaseEndpoint {
|
||||
};
|
||||
|
||||
[SuppressMessage("ReSharper", "ConvertToLambdaExpression")]
|
||||
private static IEnumerable<Attachment> ReadAttachments(JsonElement.ArrayEnumerator array, string path) => array.Select(ele => new Attachment {
|
||||
Id = ele.RequireSnowflake("id", path),
|
||||
Name = ele.RequireString("name", path),
|
||||
Type = ele.HasKey("type") ? ele.RequireString("type", path) : null,
|
||||
Url = ele.RequireString("url", path),
|
||||
Size = (ulong) ele.RequireLong("size", path),
|
||||
Width = ele.HasKey("width") ? ele.RequireInt("width", path) : null,
|
||||
Height = ele.HasKey("height") ? ele.RequireInt("height", path) : null,
|
||||
private static IEnumerable<Attachment> ReadAttachments(JsonElement.ArrayEnumerator array, string path) => array.Select(ele => {
|
||||
var downloadUrl = ele.RequireString("url", path);
|
||||
return new Attachment {
|
||||
Id = ele.RequireSnowflake("id", path),
|
||||
Name = ele.RequireString("name", path),
|
||||
Type = ele.HasKey("type") ? ele.RequireString("type", path) : null,
|
||||
NormalizedUrl = DiscordCdn.NormalizeUrl(downloadUrl),
|
||||
DownloadUrl = downloadUrl,
|
||||
Size = (ulong) ele.RequireLong("size", path),
|
||||
Width = ele.HasKey("width") ? ele.RequireInt("width", path) : null,
|
||||
Height = ele.HasKey("height") ? ele.RequireInt("height", path) : null,
|
||||
};
|
||||
}).DistinctByKeyStable(static attachment => {
|
||||
// Some Discord messages have duplicate attachments with the same id for unknown reasons.
|
||||
return attachment.Id;
|
||||
|
@@ -3,14 +3,13 @@ using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using DHT.Server.Data;
|
||||
using DHT.Server.Database;
|
||||
using DHT.Server.Service;
|
||||
using DHT.Utils.Http;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
|
||||
namespace DHT.Server.Endpoints;
|
||||
|
||||
sealed class TrackUsersEndpoint : BaseEndpoint {
|
||||
public TrackUsersEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db, parameters) {}
|
||||
public TrackUsersEndpoint(IDatabaseFile db) : base(db) {}
|
||||
|
||||
protected override async Task<IHttpOutput> Respond(HttpContext ctx) {
|
||||
var root = await ReadJson(ctx);
|
||||
|
@@ -1,30 +1,43 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<RootNamespace>DHT.Server</RootNamespace>
|
||||
<Nullable>enable</Nullable>
|
||||
<AssemblyName>DiscordHistoryTracker.Server</AssemblyName>
|
||||
<PackageId>DiscordHistoryTrackerServer</PackageId>
|
||||
<Authors>chylex</Authors>
|
||||
<Company>DiscordHistoryTracker</Company>
|
||||
<Product>DiscordHistoryTrackerServer</Product>
|
||||
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
|
||||
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
|
||||
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
|
||||
</PropertyGroup>
|
||||
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
|
||||
<DebugSymbols>true</DebugSymbols>
|
||||
<DebugType>none</DebugType>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<FrameworkReference Include="Microsoft.AspNetCore.App" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Data.Sqlite" Version="7.0.9" />
|
||||
<PackageReference Include="Microsoft.Data.Sqlite" Version="8.0.0" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\Utils\Utils.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<Compile Include="..\Version.cs" Link="Version.cs" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<EmbeddedResource Include="../Resources/Tracker/bootstrap.js">
|
||||
<LogicalName>Tracker\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
|
||||
<Link>Resources/Tracker/%(RecursiveDir)%(Filename)%(Extension)</Link>
|
||||
<Visible>false</Visible>
|
||||
</EmbeddedResource>
|
||||
<EmbeddedResource Include="../Resources/Tracker/scripts/**">
|
||||
<LogicalName>Tracker\scripts\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
|
||||
<Link>Resources/Tracker/scripts/%(RecursiveDir)%(Filename)%(Extension)</Link>
|
||||
<Visible>false</Visible>
|
||||
</EmbeddedResource>
|
||||
<EmbeddedResource Include="../Resources/Tracker/styles/**">
|
||||
<LogicalName>Tracker\styles\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
|
||||
<Link>Resources/Tracker/styles/%(RecursiveDir)%(Filename)%(Extension)</Link>
|
||||
<Visible>false</Visible>
|
||||
</EmbeddedResource>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
@@ -0,0 +1,44 @@
|
||||
using System.Net;
|
||||
using System.Threading.Tasks;
|
||||
using DHT.Utils.Logging;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.Extensions.Primitives;
|
||||
|
||||
namespace DHT.Server.Service.Middlewares;
|
||||
|
||||
sealed class ServerAuthorizationMiddleware {
|
||||
private static readonly Log Log = Log.ForType<ServerAuthorizationMiddleware>();
|
||||
|
||||
private readonly RequestDelegate next;
|
||||
private readonly ServerParameters serverParameters;
|
||||
|
||||
public ServerAuthorizationMiddleware(RequestDelegate next, ServerParameters serverParameters) {
|
||||
this.next = next;
|
||||
this.serverParameters = serverParameters;
|
||||
}
|
||||
|
||||
public async Task InvokeAsync(HttpContext context) {
|
||||
var request = context.Request;
|
||||
|
||||
bool success = HttpMethods.IsGet(request.Method)
|
||||
? CheckToken(request.Query["token"])
|
||||
: CheckToken(request.Headers["X-DHT-Token"]);
|
||||
|
||||
if (success) {
|
||||
await next(context);
|
||||
}
|
||||
else {
|
||||
context.Response.StatusCode = (int) HttpStatusCode.Forbidden;
|
||||
}
|
||||
}
|
||||
|
||||
private bool CheckToken(StringValues token) {
|
||||
if (token.Count == 1 && token[0] == serverParameters.Token) {
|
||||
return true;
|
||||
}
|
||||
else {
|
||||
Log.Error("Invalid token: " + (token.Count == 1 ? token[0] : "<missing>"));
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
29
app/Server/Service/Middlewares/ServerLoggingMiddleware.cs
Normal file
29
app/Server/Service/Middlewares/ServerLoggingMiddleware.cs
Normal file
@@ -0,0 +1,29 @@
|
||||
using System.Diagnostics;
|
||||
using System.Threading.Tasks;
|
||||
using DHT.Utils.Logging;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Http.Extensions;
|
||||
|
||||
namespace DHT.Server.Service.Middlewares;
|
||||
|
||||
sealed class ServerLoggingMiddleware {
|
||||
private static readonly Log Log = Log.ForType<ServerLoggingMiddleware>();
|
||||
|
||||
private readonly RequestDelegate next;
|
||||
|
||||
public ServerLoggingMiddleware(RequestDelegate next) {
|
||||
this.next = next;
|
||||
}
|
||||
|
||||
public async Task InvokeAsync(HttpContext context) {
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
await next(context);
|
||||
stopwatch.Stop();
|
||||
|
||||
var request = context.Request;
|
||||
var requestLength = request.ContentLength ?? 0L;
|
||||
var responseStatus = context.Response.StatusCode;
|
||||
var elapsedMs = stopwatch.ElapsedMilliseconds;
|
||||
Log.Debug("Request to " + request.GetEncodedPathAndQuery() + " (" + requestLength + " B) returned " + responseStatus + ", took " + elapsedMs + " ms");
|
||||
}
|
||||
}
|
@@ -4,7 +4,6 @@ using System.Diagnostics.CodeAnalysis;
|
||||
using System.Threading;
|
||||
using DHT.Server.Database;
|
||||
using DHT.Utils.Logging;
|
||||
using Microsoft.AspNetCore;
|
||||
using Microsoft.AspNetCore.Hosting;
|
||||
using Microsoft.AspNetCore.Server.Kestrel.Core;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
@@ -61,14 +60,12 @@ public static class ServerLauncher {
|
||||
}
|
||||
}
|
||||
|
||||
private static void StartServerFromManagementThread(int port, string token, IDatabaseFile db) {
|
||||
private static void StartServerFromManagementThread(ushort port, string token, IDatabaseFile db) {
|
||||
Log.Info("Starting server on port " + port + "...");
|
||||
|
||||
void AddServices(IServiceCollection services) {
|
||||
services.AddSingleton(typeof(IDatabaseFile), db);
|
||||
services.AddSingleton(typeof(ServerParameters), new ServerParameters {
|
||||
Token = token
|
||||
});
|
||||
services.AddSingleton(typeof(ServerParameters), new ServerParameters(port, token));
|
||||
}
|
||||
|
||||
void SetKestrelOptions(KestrelServerOptions options) {
|
||||
@@ -77,11 +74,11 @@ public static class ServerLauncher {
|
||||
options.ListenLocalhost(port, static listenOptions => listenOptions.Protocols = HttpProtocols.Http1);
|
||||
}
|
||||
|
||||
Server = WebHost.CreateDefaultBuilder()
|
||||
.ConfigureServices(AddServices)
|
||||
.UseKestrel(SetKestrelOptions)
|
||||
.UseStartup<Startup>()
|
||||
.Build();
|
||||
Server = new WebHostBuilder()
|
||||
.ConfigureServices(AddServices)
|
||||
.UseKestrel(SetKestrelOptions)
|
||||
.UseStartup<Startup>()
|
||||
.Build();
|
||||
|
||||
Server.Start();
|
||||
|
||||
@@ -103,7 +100,7 @@ public static class ServerLauncher {
|
||||
}
|
||||
}
|
||||
|
||||
public static void Relaunch(int port, string token, IDatabaseFile db) {
|
||||
public static void Relaunch(ushort port, string token, IDatabaseFile db) {
|
||||
EnqueueMessage(new IMessage.StartServer(port, token, db));
|
||||
}
|
||||
|
||||
@@ -113,11 +110,11 @@ public static class ServerLauncher {
|
||||
|
||||
private interface IMessage {
|
||||
public sealed class StartServer : IMessage {
|
||||
public int Port { get; }
|
||||
public ushort Port { get; }
|
||||
public string Token { get; }
|
||||
public IDatabaseFile Db { get; }
|
||||
|
||||
public StartServer(int port, string token, IDatabaseFile db) {
|
||||
public StartServer(ushort port, string token, IDatabaseFile db) {
|
||||
this.Port = port;
|
||||
this.Token = token;
|
||||
this.Db = db;
|
||||
|
@@ -1,5 +1,3 @@
|
||||
namespace DHT.Server.Service;
|
||||
|
||||
readonly struct ServerParameters {
|
||||
public string Token { get; init; }
|
||||
}
|
||||
sealed record ServerParameters(ushort Port, string Token);
|
||||
|
@@ -2,6 +2,7 @@ using System.Diagnostics.CodeAnalysis;
|
||||
using System.Text.Json.Serialization;
|
||||
using DHT.Server.Database;
|
||||
using DHT.Server.Endpoints;
|
||||
using DHT.Server.Service.Middlewares;
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http.Json;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
@@ -15,6 +16,7 @@ sealed class Startup {
|
||||
"https://ptb.discord.com",
|
||||
"https://canary.discord.com",
|
||||
"https://discordapp.com",
|
||||
"null" // For file:// protocol in the Viewer
|
||||
};
|
||||
|
||||
public void ConfigureServices(IServiceCollection services) {
|
||||
@@ -24,27 +26,27 @@ sealed class Startup {
|
||||
|
||||
services.AddCors(static cors => {
|
||||
cors.AddDefaultPolicy(static builder => {
|
||||
builder.WithOrigins(AllowedOrigins).AllowCredentials().AllowAnyMethod().AllowAnyHeader();
|
||||
builder.WithOrigins(AllowedOrigins).AllowCredentials().AllowAnyMethod().AllowAnyHeader().WithExposedHeaders("X-DHT");
|
||||
});
|
||||
});
|
||||
|
||||
services.AddRoutingCore();
|
||||
}
|
||||
|
||||
[SuppressMessage("ReSharper", "UnusedMember.Global")]
|
||||
public void Configure(IApplicationBuilder app, IHostApplicationLifetime lifetime, IDatabaseFile db, ServerParameters parameters) {
|
||||
app.UseRouting();
|
||||
app.UseMiddleware<ServerLoggingMiddleware>();
|
||||
app.UseCors();
|
||||
app.UseMiddleware<ServerAuthorizationMiddleware>();
|
||||
app.UseRouting();
|
||||
|
||||
app.UseEndpoints(endpoints => {
|
||||
TrackChannelEndpoint trackChannel = new(db, parameters);
|
||||
endpoints.MapPost("/track-channel", async context => await trackChannel.HandlePost(context));
|
||||
|
||||
TrackUsersEndpoint trackUsers = new(db, parameters);
|
||||
endpoints.MapPost("/track-users", async context => await trackUsers.HandlePost(context));
|
||||
|
||||
TrackMessagesEndpoint trackMessages = new(db, parameters);
|
||||
endpoints.MapPost("/track-messages", async context => await trackMessages.HandlePost(context));
|
||||
|
||||
GetAttachmentEndpoint getAttachment = new(db, parameters);
|
||||
endpoints.MapGet("/get-attachment/{url}", async context => await getAttachment.HandleGet(context));
|
||||
endpoints.MapGet("/get-tracking-script", new GetTrackingScriptEndpoint(db, parameters).Handle);
|
||||
endpoints.MapGet("/get-messages", new GetMessagesEndpoint(db).Handle);
|
||||
endpoints.MapGet("/get-attachment/{url}", new GetAttachmentEndpoint(db).Handle);
|
||||
endpoints.MapPost("/track-channel", new TrackChannelEndpoint(db).Handle);
|
||||
endpoints.MapPost("/track-users", new TrackUsersEndpoint(db).Handle);
|
||||
endpoints.MapPost("/track-messages", new TrackMessagesEndpoint(db).Handle);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
@@ -1,3 +1,5 @@
|
||||
using System.Text;
|
||||
using System.Text.Json.Serialization.Metadata;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
|
||||
@@ -12,15 +14,29 @@ public static class HttpOutput {
|
||||
}
|
||||
}
|
||||
|
||||
public sealed class Json : IHttpOutput {
|
||||
private readonly object? obj;
|
||||
public sealed class Text : IHttpOutput {
|
||||
private readonly string text;
|
||||
|
||||
public Json(object? obj) {
|
||||
this.obj = obj;
|
||||
public Text(string text) {
|
||||
this.text = text;
|
||||
}
|
||||
|
||||
public Task WriteTo(HttpResponse response) {
|
||||
return response.WriteAsJsonAsync(obj);
|
||||
return response.WriteAsync(text, Encoding.UTF8);
|
||||
}
|
||||
}
|
||||
|
||||
public sealed class Json<TValue> : IHttpOutput {
|
||||
private readonly TValue value;
|
||||
private readonly JsonTypeInfo<TValue> typeInfo;
|
||||
|
||||
public Json(TValue value, JsonTypeInfo<TValue> typeInfo) {
|
||||
this.value = value;
|
||||
this.typeInfo = typeInfo;
|
||||
}
|
||||
|
||||
public Task WriteTo(HttpResponse response) {
|
||||
return response.WriteAsJsonAsync(value, typeInfo);
|
||||
}
|
||||
}
|
||||
|
||||
|
8
app/Utils/Http/JsonElementContext.cs
Normal file
8
app/Utils/Http/JsonElementContext.cs
Normal file
@@ -0,0 +1,8 @@
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace DHT.Utils.Http;
|
||||
|
||||
[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase, GenerationMode = JsonSourceGenerationMode.Default)]
|
||||
[JsonSerializable(typeof(JsonElement))]
|
||||
public sealed partial class JsonElementContext : JsonSerializerContext {}
|
13
app/Utils/Logging/WindowsConsole.cs
Normal file
13
app/Utils/Logging/WindowsConsole.cs
Normal file
@@ -0,0 +1,13 @@
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Runtime.Versioning;
|
||||
|
||||
namespace DHT.Utils.Logging;
|
||||
|
||||
[SupportedOSPlatform("windows")]
|
||||
public static partial class WindowsConsole {
|
||||
[LibraryImport("kernel32.dll", SetLastError = true)]
|
||||
public static partial void AllocConsole();
|
||||
|
||||
[LibraryImport("kernel32.dll", SetLastError = true)]
|
||||
public static partial void FreeConsole();
|
||||
}
|
@@ -1,27 +1,25 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<RootNamespace>DHT.Utils</RootNamespace>
|
||||
<Nullable>enable</Nullable>
|
||||
<AssemblyName>DiscordHistoryTracker.Utils</AssemblyName>
|
||||
<PackageId>DiscordHistoryTrackerUtils</PackageId>
|
||||
<Authors>chylex</Authors>
|
||||
<Company>DiscordHistoryTracker</Company>
|
||||
<Product>DiscordHistoryTrackerUtils</Product>
|
||||
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
|
||||
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
|
||||
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
|
||||
</PropertyGroup>
|
||||
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
|
||||
<DebugSymbols>true</DebugSymbols>
|
||||
<DebugType>none</DebugType>
|
||||
|
||||
<PropertyGroup>
|
||||
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<FrameworkReference Include="Microsoft.AspNetCore.App" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="JetBrains.Annotations" Version="2023.2.0" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<Compile Include="..\Version.cs" Link="Version.cs" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
@@ -8,5 +8,5 @@ using DHT.Utils;
|
||||
namespace DHT.Utils;
|
||||
|
||||
static class Version {
|
||||
public const string Tag = "38.0.0.0";
|
||||
public const string Tag = "39.1.0.0";
|
||||
}
|
||||
|
@@ -4,11 +4,11 @@ set list=win-x64 linux-x64 osx-x64
|
||||
rmdir /S /Q bin
|
||||
|
||||
(for %%a in (%list%) do (
|
||||
dotnet publish Desktop -c Release -r %%a -o ./bin/%%a -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true -p:PublishReadyToRun=false -p:PublishTrimmed=true -p:TrimMode=partial -p:JsonSerializerIsReflectionEnabledByDefault=true --self-contained true
|
||||
dotnet publish Desktop -c Release -r %%a -o ./bin/%%a --self-contained true
|
||||
powershell "Compress-Archive -Path ./bin/%%a/* -DestinationPath ./bin/%%a.zip -CompressionLevel Optimal"
|
||||
))
|
||||
|
||||
dotnet publish Desktop -c Release -o ./bin/portable --self-contained false
|
||||
dotnet publish Desktop -c Release -o ./bin/portable -p:PublishSingleFile=false -p:PublishTrimmed=false --self-contained false
|
||||
powershell "Compress-Archive -Path ./bin/portable/* -DestinationPath ./bin/portable.zip -CompressionLevel Optimal"
|
||||
|
||||
echo Done
|
||||
|
@@ -17,9 +17,9 @@ rm -rf "./bin"
|
||||
configurations=(win-x64 linux-x64 osx-x64)
|
||||
|
||||
for cfg in ${configurations[@]}; do
|
||||
dotnet publish Desktop -c Release -r "$cfg" -o "./bin/$cfg" -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true -p:PublishReadyToRun=false -p:PublishTrimmed=true -p:TrimMode=partial -p:JsonSerializerIsReflectionEnabledByDefault=true --self-contained true
|
||||
dotnet publish Desktop -c Release -r "$cfg" -o "./bin/$cfg" --self-contained true
|
||||
makezip "$cfg"
|
||||
done
|
||||
|
||||
dotnet publish Desktop -c Release -o "./bin/portable" --self-contained false
|
||||
dotnet publish Desktop -c Release -o "./bin/portable" -p:PublishSingleFile=false -p:PublishTrimmed=false --self-contained false
|
||||
makezip "portable"
|
||||
|
BIN
app/empty.dht
BIN
app/empty.dht
Binary file not shown.
1072
lib/NODE-LICENSE
1072
lib/NODE-LICENSE
File diff suppressed because it is too large
Load Diff
BIN
lib/node.exe
BIN
lib/node.exe
Binary file not shown.
22
lib/node_modules/commander/LICENSE
generated
vendored
22
lib/node_modules/commander/LICENSE
generated
vendored
@@ -1,22 +0,0 @@
|
||||
(The MIT License)
|
||||
|
||||
Copyright (c) 2011 TJ Holowaychuk <tj@vision-media.ca>
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of this software and associated documentation files (the
|
||||
'Software'), to deal in the Software without restriction, including
|
||||
without limitation the rights to use, copy, modify, merge, publish,
|
||||
distribute, sublicense, and/or sell copies of the Software, and to
|
||||
permit persons to whom the Software is furnished to do so, subject to
|
||||
the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be
|
||||
included in all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
|
||||
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
|
||||
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
|
||||
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
|
||||
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
1110
lib/node_modules/commander/index.js
generated
vendored
1110
lib/node_modules/commander/index.js
generated
vendored
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user