1
0
mirror of https://github.com/chylex/Discord-History-Tracker.git synced 2025-08-17 19:31:42 +02:00

30 Commits

Author SHA1 Message Date
4f5e27f651 Release v39.1 2023-12-22 16:31:11 +01:00
cbf81ec95a Fix missing JSON source generator when parsing integrated server requests 2023-12-22 16:31:11 +01:00
8a80cb8c20 Show progress dialog when upgrading database schema 2023-12-22 16:18:03 +01:00
865deb356a Fix progress dialog not propagating exceptions from its task 2023-12-22 14:47:55 +01:00
069ab97196 Disable reflection-based JSON serialization 2023-12-22 05:54:24 +01:00
caab038eaa Use source generators for JSON serialization everywhere 2023-12-22 05:24:28 +01:00
fb837374fc Enable single file compression and disable unnecessary .NET features 2023-12-22 02:30:24 +01:00
65d935cca1 Use compiled bindings in Avalonia XAML 2023-12-21 08:55:56 +01:00
6e64c86d7a Optimize viewer JSON export using source generators 2023-12-21 08:29:07 +01:00
8aeb590bb3 Release v39.0 2023-12-21 05:18:01 +01:00
8dc1adc9f0 Fix viewer converting underscores to italics even when not at the end of a word 2023-12-21 05:09:55 +01:00
ddf70b02e7 Fix not storing channel topic and position if the channel is in a category
Closes #232
2023-12-21 04:49:19 +01:00
ef59fd992e Reword "Do Nothing" option in autoscroll settings to improve clarity 2023-12-21 04:27:24 +01:00
d044627fac Reduce delay in tracker for responding to new messages 2023-12-21 04:27:24 +01:00
a624745602 Force light theme in Avalonia 2023-12-21 04:24:12 +01:00
6da3c185e5 Update Avalonia to 11.0.6 2023-12-21 04:24:01 +01:00
d4d14cab97 Update Rider project configuration files 2023-12-21 00:55:19 +01:00
095c9a061a Update attachment downloads for time-limited Discord CDN URLs
Closes #223
2023-12-21 00:27:08 +01:00
d01f9ed218 Rewrite detection of newly loaded messages 2023-12-20 13:19:11 +01:00
dd6f121059 Fix not seeing messages after a Discord update
Closes #240
2023-12-20 10:51:14 +01:00
8bba33d815 Update SQLite provider to 8.0.0 (SQLite version 3.41.2) 2023-12-20 10:42:17 +01:00
9eab8ac92a Update README section on distribution 2023-11-24 13:47:53 +01:00
fe588686fc Clean up project configuration and build scripts & fix CI builds 2023-11-23 21:45:54 +01:00
7392987165 Remove unused argument in tracker script
Closes #235
2023-11-23 05:22:33 +01:00
492dddb35d Load full tracking script from the integrated server to remove need for minification 2023-11-21 07:08:26 +01:00
d35280a6a6 Release v38.0 2023-10-29 06:42:45 +01:00
ae8a34f938 Log error when reading Discord settings fails 2023-10-29 06:42:45 +01:00
18f5823f2a Fix not seeing messages after a Discord update
Closes #226
2023-10-29 06:42:45 +01:00
37640c97b0 Update to Avalonia 11.0.5 2023-10-29 06:38:26 +01:00
3cc5c75c48 Fix JSON serialization not working in release builds 2023-10-29 06:38:26 +01:00
125 changed files with 1208 additions and 42294 deletions

View File

@@ -16,7 +16,6 @@ Fork the repository and clone it to your computer (if you've never used git, you
Folder organization:
* `app/` contains a Visual Studio solution for the desktop app
* `lib/` contains utilities required to build the project
* `web/` contains source code of the [official website](https://dht.chylex.com), which can be used as a template when making your own website
To start editing source code for the desktop app, install the [.NET 8 SDK](https://dotnet.microsoft.com/en-us/download/dotnet/8.0), and then open `app/DiscordHistoryTracker.sln` in [Visual Studio](https://visualstudio.microsoft.com/downloads/) or [Rider](https://www.jetbrains.com/rider/).
@@ -29,22 +28,13 @@ To build a `Release` version of the desktop app, follow the instructions for you
#### Release Windows (64-bit)
1. Install [Python 3](https://www.python.org/downloads), and ensure the `python` executable is in your `PATH`
2. Install [Powershell 5](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows) or newer (on Windows 10, the included version of Powershell should be enough)
The `lib/` folder contains an installation of [Node](https://nodejs.org/en) and [uglify-js](https://www.npmjs.com/package/uglify-js), which are used to minify the tracking script. This installation will only work on 64-bit Windows; building on 32-bit Windows is not supported, but you can try.
1. Install [Powershell 5](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows) or newer (on Windows 10, the included version of Powershell should be enough)
Run the `app/build.bat` script, and read the [Distribution](#distribution) section below.
#### Release Other Operating Systems
1. Install [Python 3](https://www.python.org/downloads), and ensure the `python` executable exists and launches Python 3
- On Debian and derivatives, you can install `python-is-python3`
- On other distributions, you can create a link manually, for ex. `ln -s /usr/bin/python3 /usr/bin/python`
- If you don't want `python` to mean Python 3, then edit `Desktop.csproj` and change `python` to `python3`
2. Install [Node + npm](https://nodejs.org/en)
3. Install [uglify-js](https://www.npmjs.com/package/uglify-js) globally (`npm install -g uglify-js`)
4. Install the `zip` package from your repository
1. Install the `zip` package from your repository
Run the `app/build.sh` script, and read the [Distribution](#distribution) section below.
@@ -52,4 +42,4 @@ Run the `app/build.sh` script, and read the [Distribution](#distribution) sectio
The mentioned build scripts will prepare `Release` builds ready for distribution. Once the script finishes, the `app/bin` folder will contain self-contained executables for each major operating system, and a portable version that works on all other systems but requires .NET 8 to be installed.
Note that when building on Windows, the generated `.zip` files for Linux and Mac will not have correct file permissions, so it will not be possible to run them by double-clicking `DiscordHistoryTracker`. I tried using Python to re-create the archives with correct file permissions, but found that Linux `zip` tools could not see them. The only working solution is building the Windows + portable version on Windows, and Linux + Mac version on Linux.
Note that when building on Windows, the generated `.zip` files for Linux and Mac will not have correct file permissions, so it will not be possible to run them by double-clicking the executable. Since .NET 8 fixed several issues with publishing Windows executables on Linux, I recommend using Linux to build the app for all operating systems.

View File

@@ -29,9 +29,6 @@
<H2CodeStyleSettings version="6">
<option name="USE_GENERIC_STYLE" value="true" />
</H2CodeStyleSettings>
<H2CodeStyleSettings version="6">
<option name="USE_GENERIC_STYLE" value="true" />
</H2CodeStyleSettings>
<HSQLCodeStyleSettings version="6">
<option name="USE_GENERIC_STYLE" value="true" />
</HSQLCodeStyleSettings>

View File

@@ -1,8 +1,8 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Desktop" type="DotNetProject" factoryName=".NET Project">
<option name="EXE_PATH" value="$PROJECT_DIR$/Desktop/bin/Debug/net5.0/DiscordHistoryTracker.exe" />
<option name="EXE_PATH" value="$PROJECT_DIR$/.artifacts/bin/Desktop/debug/DiscordHistoryTracker.exe" />
<option name="PROGRAM_PARAMETERS" value="" />
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$/Desktop/bin/Debug/net5.0" />
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$/.artifacts/bin/Desktop/debug" />
<option name="PASS_PARENT_ENVS" value="1" />
<option name="USE_EXTERNAL_CONSOLE" value="0" />
<option name="USE_MONO" value="0" />
@@ -12,7 +12,7 @@
<option name="PROJECT_ARGUMENTS_TRACKING" value="1" />
<option name="PROJECT_WORKING_DIRECTORY_TRACKING" value="1" />
<option name="PROJECT_KIND" value="DotNetCore" />
<option name="PROJECT_TFM" value="net5.0" />
<option name="PROJECT_TFM" value="net8.0" />
<method v="2">
<option name="Build" />
</method>

View File

@@ -1,23 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Minify" type="PythonConfigurationType" factoryName="Python">
<module name="rider.module" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="PARENT_ENVS" value="true" />
<envs>
<env name="PYTHONUNBUFFERED" value="1" />
</envs>
<option name="SDK_HOME" value="" />
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$" />
<option name="IS_MODULE_SDK" value="false" />
<option name="ADD_CONTENT_ROOTS" value="false" />
<option name="ADD_SOURCE_ROOTS" value="false" />
<option name="SCRIPT_NAME" value="$PROJECT_DIR$/minify.py" />
<option name="PARAMETERS" value="" />
<option name="SHOW_COMMAND_LINE" value="false" />
<option name="EMULATE_TERMINAL" value="false" />
<option name="MODULE_MODE" value="false" />
<option name="REDIRECT_INPUT" value="false" />
<option name="INPUT_FILE" value="" />
<method v="2" />
</configuration>
</component>

View File

@@ -2,7 +2,8 @@
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:common="clr-namespace:DHT.Desktop.Common"
xmlns:system="clr-namespace:System;assembly=System.Runtime"
x:Class="DHT.Desktop.App">
x:Class="DHT.Desktop.App"
RequestedThemeVariant="Light">
<Application.Styles>

View File

@@ -1,9 +1,11 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using Avalonia.Controls;
using Avalonia.Platform.Storage;
using Avalonia.Threading;
using DHT.Desktop.Dialogs.File;
using DHT.Desktop.Dialogs.Message;
using DHT.Server.Database;
@@ -41,11 +43,16 @@ static class DatabaseGui {
});
}
public static async Task<IDatabaseFile?> TryOpenOrCreateDatabaseFromPath(string path, Window window, Func<Task<bool>> checkCanUpgradeDatabase) {
public static async Task<IDatabaseFile?> TryOpenOrCreateDatabaseFromPath(string path, Window window, ISchemaUpgradeCallbacks schemaUpgradeCallbacks) {
var prevSynchronizationContext = SynchronizationContext.Current;
SynchronizationContext.SetSynchronizationContext(new AvaloniaSynchronizationContext());
var taskScheduler = TaskScheduler.FromCurrentSynchronizationContext();
SynchronizationContext.SetSynchronizationContext(prevSynchronizationContext);
IDatabaseFile? file = null;
try {
file = await SqliteDatabaseFile.OpenOrCreate(path, checkCanUpgradeDatabase);
file = await SqliteDatabaseFile.OpenOrCreate(path, schemaUpgradeCallbacks, taskScheduler);
} catch (InvalidDatabaseVersionException ex) {
await Dialog.ShowOk(window, "Database Error", "Database '" + Path.GetFileName(path) + "' appears to be corrupted (invalid version: " + ex.Version + ").");
} catch (DatabaseTooNewException ex) {

View File

@@ -1,36 +1,33 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<RootNamespace>DHT.Desktop</RootNamespace>
<AssemblyName>DiscordHistoryTracker</AssemblyName>
<PackageId>DiscordHistoryTracker</PackageId>
</PropertyGroup>
<PropertyGroup>
<OutputType>WinExe</OutputType>
<Nullable>enable</Nullable>
<AssemblyName>DiscordHistoryTracker</AssemblyName>
<RootNamespace>DHT.Desktop</RootNamespace>
<PackageId>DiscordHistoryTracker</PackageId>
<Authors>chylex</Authors>
<Company>DiscordHistoryTracker</Company>
<Product>DiscordHistoryTracker</Product>
<ApplicationIcon>./Resources/icon.ico</ApplicationIcon>
<AvaloniaUseCompiledBindingsByDefault>true</AvaloniaUseCompiledBindingsByDefault>
<CheckForOverflowUnderflow>true</CheckForOverflowUnderflow>
<SatelliteResourceLanguages>en</SatelliteResourceLanguages>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<DebugSymbols>true</DebugSymbols>
<DebugType>none</DebugType>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Avalonia" Version="11.0.0" />
<PackageReference Include="Avalonia.Controls.DataGrid" Version="11.0.0" />
<PackageReference Include="Avalonia.Controls.ItemsRepeater" Version="11.0.0" />
<PackageReference Include="Avalonia.Desktop" Version="11.0.0" />
<PackageReference Include="Avalonia.Diagnostics" Version="11.0.0" Condition=" '$(Configuration)' == 'Debug' " />
<PackageReference Include="Avalonia.Fonts.Inter" Version="11.0.0" />
<PackageReference Include="Avalonia.Themes.Fluent" Version="11.0.0" />
<PackageReference Include="Avalonia" Version="11.0.6" />
<PackageReference Include="Avalonia.Controls.DataGrid" Version="11.0.6" />
<PackageReference Include="Avalonia.Controls.ItemsRepeater" Version="11.0.6" />
<PackageReference Include="Avalonia.Desktop" Version="11.0.6" />
<PackageReference Include="Avalonia.Diagnostics" Version="11.0.6" Condition=" '$(Configuration)' == 'Debug' " />
<PackageReference Include="Avalonia.Fonts.Inter" Version="11.0.6" />
<PackageReference Include="Avalonia.Themes.Fluent" Version="11.0.6" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\Server\Server.csproj" />
</ItemGroup>
<ItemGroup>
<Compile Include="..\Version.cs" Link="Version.cs" />
<Compile Update="Dialogs\TextBox\TextBoxDialog.axaml.cs">
@@ -38,22 +35,11 @@
<SubType>Code</SubType>
</Compile>
</ItemGroup>
<ItemGroup>
<AvaloniaResource Include="Resources/icon.ico" />
<EmbeddedResource Include="../Resources/Tracker/bootstrap.js">
<LogicalName>Tracker\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
</EmbeddedResource>
<EmbeddedResource Include="../Resources/Tracker/scripts/**" Condition=" '$(Configuration)' == 'Debug' ">
<LogicalName>Tracker\scripts\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/scripts/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
</EmbeddedResource>
<EmbeddedResource Include="../Resources/Tracker/styles/**">
<LogicalName>Tracker\styles\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/styles/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
<EmbeddedResource Include="Resources/tracker-loader.js">
<LogicalName>tracker-loader.js</LogicalName>
</EmbeddedResource>
<EmbeddedResource Include="../Resources/Viewer/**">
<LogicalName>Viewer\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
@@ -61,19 +47,5 @@
<Visible>false</Visible>
</EmbeddedResource>
</ItemGroup>
<Target Name="MinifyResources" BeforeTargets="PrepareForBuild" Condition=" '$(Configuration)' == 'Release' ">
<PropertyGroup>
<MinifiedResourceDir>$(ProjectDir)bin/.res/scripts</MinifiedResourceDir>
</PropertyGroup>
<ItemGroup>
<UpToDateCheckInput Include="$(ProjectDir)../Resources/Tracker/scripts/**" Visible="false" />
<EmbeddedResource Include="$(MinifiedResourceDir)/discord.js" LogicalName="Tracker\scripts\discord.js" Visible="false" />
<EmbeddedResource Include="$(MinifiedResourceDir)/dom.js" LogicalName="Tracker\scripts\dom.js" Visible="false" />
<EmbeddedResource Include="$(MinifiedResourceDir)/gui.js" LogicalName="Tracker\scripts\gui.js" Visible="false" />
<EmbeddedResource Include="$(MinifiedResourceDir)/settings.js" LogicalName="Tracker\scripts\settings.js" Visible="false" />
<EmbeddedResource Include="$(MinifiedResourceDir)/state.js" LogicalName="Tracker\scripts\state.js" Visible="false" />
</ItemGroup>
<RemoveDir Directories="$(ProjectDir)bin/.res/scripts" />
<Exec Command="python $(ProjectDir)../Resources/minify.py" WorkingDirectory="$(ProjectDir)../Resources" IgnoreExitCode="false" />
</Target>
</Project>

View File

@@ -5,6 +5,7 @@
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.CheckBox"
mc:Ignorable="d" d:DesignWidth="500"
x:Class="DHT.Desktop.Dialogs.CheckBox.CheckBoxDialog"
x:DataType="namespace:CheckBoxDialogModel"
Title="{Binding Title}"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Width="500" SizeToContent="Height" CanResize="False"

View File

@@ -5,6 +5,7 @@
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.Message"
mc:Ignorable="d" d:DesignWidth="500"
x:Class="DHT.Desktop.Dialogs.Message.MessageDialog"
x:DataType="namespace:MessageDialogModel"
Title="{Binding Title}"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Width="500" SizeToContent="Height" CanResize="False"

View File

@@ -4,4 +4,5 @@ namespace DHT.Desktop.Dialogs.Progress;
interface IProgressCallback {
Task Update(string message, int finishedItems, int totalItems);
Task Hide();
}

View File

@@ -5,6 +5,7 @@
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.Progress"
mc:Ignorable="d" d:DesignWidth="500"
x:Class="DHT.Desktop.Dialogs.Progress.ProgressDialog"
x:DataType="namespace:ProgressDialogModel"
Title="{Binding Title}"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Opened="OnOpened"
@@ -31,12 +32,18 @@
</Style>
</Window.Styles>
<StackPanel Margin="20">
<ItemsRepeater ItemsSource="{Binding Items}" Margin="0 10">
<ItemsRepeater.ItemTemplate>
<DataTemplate>
<StackPanel Margin="20 10" IsHitTestVisible="{Binding IsVisible}" Opacity="{Binding Opacity}">
<DockPanel>
<TextBlock DockPanel.Dock="Right" Text="{Binding Items}" Classes="items" />
<TextBlock DockPanel.Dock="Left" Text="{Binding Message}" />
</DockPanel>
<ProgressBar Value="{Binding Progress}" />
</StackPanel>
</DataTemplate>
</ItemsRepeater.ItemTemplate>
</ItemsRepeater>
</Window>

View File

@@ -8,6 +8,7 @@ namespace DHT.Desktop.Dialogs.Progress;
[SuppressMessage("ReSharper", "MemberCanBeInternal")]
public sealed partial class ProgressDialog : Window {
private bool isFinished = false;
private Task progressTask = Task.CompletedTask;
public ProgressDialog() {
InitializeComponent();
@@ -15,7 +16,8 @@ public sealed partial class ProgressDialog : Window {
public void OnOpened(object? sender, EventArgs e) {
if (DataContext is ProgressDialogModel model) {
Task.Run(model.StartTask).ContinueWith(OnFinished, TaskScheduler.FromCurrentSynchronizationContext());
progressTask = Task.Run(model.StartTask);
progressTask.ContinueWith(OnFinished, TaskScheduler.FromCurrentSynchronizationContext());
}
}
@@ -27,4 +29,9 @@ public sealed partial class ProgressDialog : Window {
isFinished = true;
Close();
}
public async Task ShowProgressDialog(Window owner) {
await ShowDialog(owner);
await progressTask;
}
}

View File

@@ -1,4 +1,6 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Avalonia.Threading;
using DHT.Desktop.Common;
@@ -9,57 +11,43 @@ namespace DHT.Desktop.Dialogs.Progress;
sealed class ProgressDialogModel : BaseModel {
public string Title { get; init; } = "";
private string message = "";
public string Message {
get => message;
private set => Change(ref message, value);
}
private string items = "";
public string Items {
get => items;
private set => Change(ref items, value);
}
private int progress = 0;
public int Progress {
get => progress;
private set => Change(ref progress, value);
}
public IReadOnlyList<ProgressItem> Items { get; } = Array.Empty<ProgressItem>();
private readonly TaskRunner? task;
[Obsolete("Designer")]
public ProgressDialogModel() {}
public ProgressDialogModel(TaskRunner task) {
public ProgressDialogModel(TaskRunner task, int progressItems = 1) {
this.Items = Enumerable.Range(0, progressItems).Select(static _ => new ProgressItem()).ToArray();
this.task = task;
}
internal async Task StartTask() {
if (task != null) {
await task(new Callback(this));
await task(Items.Select(static item => new Callback(item)).ToArray());
}
}
public delegate Task TaskRunner(IProgressCallback callback);
public delegate Task TaskRunner(IReadOnlyList<IProgressCallback> callbacks);
private sealed class Callback : IProgressCallback {
private readonly ProgressDialogModel model;
private readonly ProgressItem item;
public Callback(ProgressDialogModel model) {
this.model = model;
public Callback(ProgressItem item) {
this.item = item;
}
async Task IProgressCallback.Update(string message, int finishedItems, int totalItems) {
public async Task Update(string message, int finishedItems, int totalItems) {
await Dispatcher.UIThread.InvokeAsync(() => {
model.Message = message;
model.Items = finishedItems.Format() + " / " + totalItems.Format();
model.Progress = 100 * finishedItems / totalItems;
item.Message = message;
item.Items = totalItems == 0 ? string.Empty : finishedItems.Format() + " / " + totalItems.Format();
item.Progress = totalItems == 0 ? 0 : 100 * finishedItems / totalItems;
});
}
public Task Hide() {
return Update(string.Empty, 0, 0);
}
}
}

View File

@@ -0,0 +1,41 @@
using DHT.Utils.Models;
namespace DHT.Desktop.Dialogs.Progress;
sealed class ProgressItem : BaseModel {
private bool isVisible = false;
public bool IsVisible {
get => isVisible;
private set {
Change(ref isVisible, value);
OnPropertyChanged(nameof(Opacity));
}
}
public double Opacity => IsVisible ? 1.0 : 0.0;
private string message = "";
public string Message {
get => message;
set {
Change(ref message, value);
IsVisible = !string.IsNullOrEmpty(value);
}
}
private string items = "";
public string Items {
get => items;
set => Change(ref items, value);
}
private int progress = 0;
public int Progress {
get => progress;
set => Change(ref progress, value);
}
}

View File

@@ -5,6 +5,7 @@
xmlns:namespace="clr-namespace:DHT.Desktop.Dialogs.TextBox"
mc:Ignorable="d" d:DesignWidth="500"
x:Class="DHT.Desktop.Dialogs.TextBox.TextBoxDialog"
x:DataType="namespace:TextBoxDialogModel"
Title="{Binding Title}"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Width="500" SizeToContent="Height" CanResize="False"

View File

@@ -1,9 +1,9 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.IO;
using System.Runtime.InteropServices;
using System.Text.Json;
using System.Text.Json.Nodes;
using System.Threading.Tasks;
using DHT.Utils.Logging;
using static System.Environment.SpecialFolder;
@@ -40,17 +40,19 @@ static class DiscordAppSettings {
public static async Task<bool?> AreDevToolsEnabled() {
try {
return AreDevToolsEnabled(await ReadSettingsJson());
} catch (Exception) {
} catch (Exception e) {
Log.Error("Cannot read settings file.");
Log.Error(e);
return null;
}
}
private static bool AreDevToolsEnabled(Dictionary<string, object?> json) {
return json.TryGetValue(JsonKeyDevTools, out var value) && value is JsonElement { ValueKind: JsonValueKind.True };
private static bool AreDevToolsEnabled(JsonObject json) {
return json.TryGetPropertyValue(JsonKeyDevTools, out var node) && node?.GetValueKind() == JsonValueKind.True;
}
public static async Task<SettingsJsonResult> ConfigureDevTools(bool enable) {
Dictionary<string, object?> json;
JsonObject json;
try {
json = await ReadSettingsJson();
@@ -107,13 +109,13 @@ static class DiscordAppSettings {
return SettingsJsonResult.Success;
}
private static async Task<Dictionary<string, object?>> ReadSettingsJson() {
private static async Task<JsonObject> ReadSettingsJson() {
await using var stream = new FileStream(JsonFilePath, FileMode.Open, FileAccess.Read, FileShare.Read);
return await JsonSerializer.DeserializeAsync<Dictionary<string, object?>?>(stream) ?? throw new JsonException();
return await JsonSerializer.DeserializeAsync(stream, DiscordAppSettingsJsonContext.Default.JsonObject) ?? throw new JsonException();
}
private static async Task WriteSettingsJson(Dictionary<string, object?> json) {
private static async Task WriteSettingsJson(JsonObject json) {
await using var stream = new FileStream(JsonFilePath, FileMode.Truncate, FileAccess.Write, FileShare.None);
await JsonSerializer.SerializeAsync(stream, json, new JsonSerializerOptions { WriteIndented = true });
await JsonSerializer.SerializeAsync(stream, json, DiscordAppSettingsJsonContext.Default.JsonObject);
}
}

View File

@@ -0,0 +1,8 @@
using System.Text.Json.Nodes;
using System.Text.Json.Serialization;
namespace DHT.Desktop.Discord;
[JsonSourceGenerationOptions(GenerationMode = JsonSourceGenerationMode.Default, WriteIndented = true)]
[JsonSerializable(typeof(JsonObject))]
sealed partial class DiscordAppSettingsJsonContext : JsonSerializerContext {}

View File

@@ -5,6 +5,7 @@
xmlns:main="clr-namespace:DHT.Desktop.Main"
mc:Ignorable="d" d:DesignWidth="480" d:DesignHeight="295"
x:Class="DHT.Desktop.Main.AboutWindow"
x:DataType="main:AboutWindowModel"
Title="About Discord History Tracker"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Width="480" Height="295" CanResize="False"

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d"
x:Class="DHT.Desktop.Main.Controls.AttachmentFilterPanel">
x:Class="DHT.Desktop.Main.Controls.AttachmentFilterPanel"
x:DataType="controls:AttachmentFilterPanelModel">
<Design.DataContext>
<controls:AttachmentFilterPanelModel />

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d"
x:Class="DHT.Desktop.Main.Controls.MessageFilterPanel">
x:Class="DHT.Desktop.Main.Controls.MessageFilterPanel"
x:DataType="controls:MessageFilterPanelModel">
<Design.DataContext>
<controls:MessageFilterPanelModel />

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d"
x:Class="DHT.Desktop.Main.Controls.ServerConfigurationPanel">
x:Class="DHT.Desktop.Main.Controls.ServerConfigurationPanel"
x:DataType="controls:ServerConfigurationPanelModel">
<Design.DataContext>
<controls:ServerConfigurationPanelModel />

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d"
x:Class="DHT.Desktop.Main.Controls.StatusBar">
x:Class="DHT.Desktop.Main.Controls.StatusBar"
x:DataType="controls:StatusBarModel">
<Design.DataContext>
<controls:StatusBarModel />

View File

@@ -5,6 +5,7 @@
xmlns:main="clr-namespace:DHT.Desktop.Main"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.MainWindow"
x:DataType="main:MainWindowModel"
Title="{Binding Title}"
Icon="avares://DiscordHistoryTracker/Resources/icon.ico"
Width="800" Height="500"

View File

@@ -5,7 +5,8 @@
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.AdvancedPage">
x:Class="DHT.Desktop.Main.Pages.AdvancedPage"
x:DataType="pages:AdvancedPageModel">
<Design.DataContext>
<pages:AdvancedPageModel />

View File

@@ -5,7 +5,8 @@
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.AttachmentsPage">
x:Class="DHT.Desktop.Main.Pages.AttachmentsPage"
x:DataType="pages:AttachmentsPageModel">
<Design.DataContext>
<pages:AttachmentsPageModel />
@@ -35,7 +36,7 @@
<TextBlock Text="{Binding DownloadMessage}" Margin="10 0 0 0" VerticalAlignment="Center" DockPanel.Dock="Left" />
<ProgressBar Value="{Binding DownloadProgress}" IsVisible="{Binding IsDownloading}" Margin="15 0" VerticalAlignment="Center" DockPanel.Dock="Right" />
</DockPanel>
<controls:AttachmentFilterPanel DataContext="{Binding FilterModel}" IsEnabled="{Binding !DataContext.IsDownloading, RelativeSource={RelativeSource AncestorType=UserControl}}" />
<controls:AttachmentFilterPanel DataContext="{Binding FilterModel}" IsEnabled="{Binding !IsDownloading, RelativeSource={RelativeSource AncestorType=pages:AttachmentsPageModel}}" />
<StackPanel Orientation="Vertical" Spacing="12">
<Expander Header="Download Status" IsExpanded="True">
<DataGrid ItemsSource="{Binding StatisticsRows}" AutoGenerateColumns="False" CanUserReorderColumns="False" CanUserResizeColumns="False" CanUserSortColumns="False" IsReadOnly="True">

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.DatabasePage">
x:Class="DHT.Desktop.Main.Pages.DatabasePage"
x:DataType="pages:DatabasePageModel">
<Design.DataContext>
<pages:DatabasePageModel />

View File

@@ -17,6 +17,7 @@ using DHT.Desktop.Dialogs.TextBox;
using DHT.Server.Data;
using DHT.Server.Database;
using DHT.Server.Database.Import;
using DHT.Server.Database.Sqlite;
using DHT.Utils.Logging;
using DHT.Utils.Models;
@@ -77,31 +78,18 @@ sealed class DatabasePageModel : BaseModel {
}
ProgressDialog progressDialog = new ProgressDialog();
progressDialog.DataContext = new ProgressDialogModel(async callback => await MergeWithDatabaseFromPaths(Db, paths, progressDialog, callback)) {
progressDialog.DataContext = new ProgressDialogModel(async callbacks => await MergeWithDatabaseFromPaths(Db, paths, progressDialog, callbacks[0])) {
Title = "Database Merge"
};
await progressDialog.ShowDialog(window);
await progressDialog.ShowProgressDialog(window);
}
private static async Task MergeWithDatabaseFromPaths(IDatabaseFile target, string[] paths, ProgressDialog dialog, IProgressCallback callback) {
int total = paths.Length;
DialogResult.YesNo? upgradeResult = null;
async Task<bool> CheckCanUpgradeDatabase() {
upgradeResult ??= total > 1
? await DatabaseGui.ShowCanUpgradeMultipleDatabaseDialog(dialog)
: await DatabaseGui.ShowCanUpgradeDatabaseDialog(dialog);
return DialogResult.YesNo.Yes == upgradeResult;
}
var schemaUpgradeCallbacks = new SchemaUpgradeCallbacks(dialog, paths.Length);
await PerformImport(target, paths, dialog, callback, "Database Merge", "Database Error", "database file", async path => {
SynchronizationContext? prevSyncContext = SynchronizationContext.Current;
SynchronizationContext.SetSynchronizationContext(new AvaloniaSynchronizationContext());
IDatabaseFile? db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, dialog, CheckCanUpgradeDatabase);
SynchronizationContext.SetSynchronizationContext(prevSyncContext);
IDatabaseFile? db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, dialog, schemaUpgradeCallbacks);
if (db == null) {
return false;
@@ -116,6 +104,41 @@ sealed class DatabasePageModel : BaseModel {
});
}
private sealed class SchemaUpgradeCallbacks : ISchemaUpgradeCallbacks {
private readonly ProgressDialog dialog;
private readonly int total;
private bool? decision;
public SchemaUpgradeCallbacks(ProgressDialog dialog, int total) {
this.total = total;
this.dialog = dialog;
}
public async Task<bool> CanUpgrade() {
return decision ??= (total > 1
? await DatabaseGui.ShowCanUpgradeMultipleDatabaseDialog(dialog)
: await DatabaseGui.ShowCanUpgradeDatabaseDialog(dialog)) == DialogResult.YesNo.Yes;
}
public Task Start(int versionSteps, Func<ISchemaUpgradeCallbacks.IProgressReporter, Task> doUpgrade) {
return doUpgrade(new NullReporter());
}
private sealed class NullReporter : ISchemaUpgradeCallbacks.IProgressReporter {
public Task NextVersion() {
return Task.CompletedTask;
}
public Task MainWork(string message, int finishedItems, int totalItems) {
return Task.CompletedTask;
}
public Task SubWork(string message, int finishedItems, int totalItems) {
return Task.CompletedTask;
}
}
}
public async void ImportLegacyArchive() {
var paths = await window.StorageProvider.OpenFiles(new FilePickerOpenOptions {
Title = "Open Legacy DHT Archive",
@@ -128,11 +151,11 @@ sealed class DatabasePageModel : BaseModel {
}
ProgressDialog progressDialog = new ProgressDialog();
progressDialog.DataContext = new ProgressDialogModel(async callback => await ImportLegacyArchiveFromPaths(Db, paths, progressDialog, callback)) {
progressDialog.DataContext = new ProgressDialogModel(async callbacks => await ImportLegacyArchiveFromPaths(Db, paths, progressDialog, callbacks[0])) {
Title = "Legacy Archive Import"
};
await progressDialog.ShowDialog(window);
await progressDialog.ShowProgressDialog(window);
}
private static async Task ImportLegacyArchiveFromPaths(IDatabaseFile target, string[] paths, ProgressDialog dialog, IProgressCallback callback) {

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.DebugPage">
x:Class="DHT.Desktop.Main.Pages.DebugPage"
x:DataType="pages:DebugPageModel">
<Design.DataContext>
<pages:DebugPageModel />

View File

@@ -45,12 +45,12 @@ namespace DHT.Desktop.Main.Pages {
}
ProgressDialog progressDialog = new ProgressDialog {
DataContext = new ProgressDialogModel(async callback => await GenerateRandomData(channels, users, messages, callback)) {
DataContext = new ProgressDialogModel(async callbacks => await GenerateRandomData(channels, users, messages, callbacks[0])) {
Title = "Generating Random Data"
}
};
await progressDialog.ShowDialog(window);
await progressDialog.ShowProgressDialog(window);
}
private const int BatchSize = 500;

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.TrackingPage">
x:Class="DHT.Desktop.Main.Pages.TrackingPage"
x:DataType="pages:TrackingPageModel">
<Design.DataContext>
<pages:TrackingPageModel />

View File

@@ -54,12 +54,8 @@ sealed class TrackingPageModel : BaseModel {
}
public async Task<bool> OnClickCopyTrackingScript() {
string bootstrap = await Resources.ReadTextAsync("Tracker/bootstrap.js");
string script = bootstrap.Replace("= 0; /*[PORT]*/", "= " + ServerManager.Port + ";")
.Replace("/*[TOKEN]*/", HttpUtility.JavaScriptStringEncode(ServerManager.Token))
.Replace("/*[IMPORTS]*/", await Resources.ReadJoinedAsync("Tracker/scripts/", '\n'))
.Replace("/*[CSS-CONTROLLER]*/", await Resources.ReadTextAsync("Tracker/styles/controller.css"))
.Replace("/*[CSS-SETTINGS]*/", await Resources.ReadTextAsync("Tracker/styles/settings.css"));
string url = $"http://127.0.0.1:{ServerManager.Port}/get-tracking-script?token={HttpUtility.UrlEncode(ServerManager.Token)}";
string script = (await Resources.ReadTextAsync("tracker-loader.js")).Trim().Replace("{url}", url);
var clipboard = window.Clipboard;
if (clipboard == null) {

View File

@@ -5,7 +5,8 @@
xmlns:pages="clr-namespace:DHT.Desktop.Main.Pages"
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Pages.ViewerPage">
x:Class="DHT.Desktop.Main.Pages.ViewerPage"
x:DataType="pages:ViewerPageModel">
<Design.DataContext>
<pages:ViewerPageModel />

View File

@@ -35,7 +35,7 @@ sealed class ViewerPageModel : BaseModel, IDisposable {
set => Change(ref hasFilters, value);
}
private MessageFilterPanelModel FilterModel { get; }
public MessageFilterPanelModel FilterModel { get; }
private readonly Window window;
private readonly IDatabaseFile db;

View File

@@ -5,7 +5,8 @@
xmlns:controls="clr-namespace:DHT.Desktop.Main.Controls"
xmlns:screens="clr-namespace:DHT.Desktop.Main.Screens"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Screens.MainContentScreen">
x:Class="DHT.Desktop.Main.Screens.MainContentScreen"
x:DataType="screens:MainContentScreenModel">
<Design.DataContext>
<screens:MainContentScreenModel />

View File

@@ -4,7 +4,8 @@
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:screens="clr-namespace:DHT.Desktop.Main.Screens"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="DHT.Desktop.Main.Screens.WelcomeScreen">
x:Class="DHT.Desktop.Main.Screens.WelcomeScreen"
x:DataType="screens:WelcomeScreenModel">
<Design.DataContext>
<screens:WelcomeScreenModel />

View File

@@ -1,10 +1,13 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
using Avalonia.Controls;
using DHT.Desktop.Common;
using DHT.Desktop.Dialogs.Message;
using DHT.Desktop.Dialogs.Progress;
using DHT.Server.Database;
using DHT.Server.Database.Sqlite;
using DHT.Utils.Models;
namespace DHT.Desktop.Main.Screens;
@@ -39,16 +42,73 @@ sealed class WelcomeScreenModel : BaseModel, IDisposable {
}
dbFilePath = path;
Db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, window, CheckCanUpgradeDatabase);
Db = await DatabaseGui.TryOpenOrCreateDatabaseFromPath(path, window, new SchemaUpgradeCallbacks(window));
OnPropertyChanged(nameof(Db));
OnPropertyChanged(nameof(HasDatabase));
}
private async Task<bool> CheckCanUpgradeDatabase() {
private sealed class SchemaUpgradeCallbacks : ISchemaUpgradeCallbacks {
private readonly Window window;
public SchemaUpgradeCallbacks(Window window) {
this.window = window;
}
public async Task<bool> CanUpgrade() {
return DialogResult.YesNo.Yes == await DatabaseGui.ShowCanUpgradeDatabaseDialog(window);
}
public async Task Start(int versionSteps, Func<ISchemaUpgradeCallbacks.IProgressReporter, Task> doUpgrade) {
async Task StartUpgrade(IReadOnlyList<IProgressCallback> callbacks) {
var reporter = new ProgressReporter(versionSteps, callbacks);
await reporter.NextVersion();
await Task.Delay(TimeSpan.FromMilliseconds(800));
await doUpgrade(reporter);
await Task.Delay(TimeSpan.FromMilliseconds(600));
}
await new ProgressDialog {
DataContext = new ProgressDialogModel(StartUpgrade, progressItems: 3) {
Title = "Upgrading Database"
}
}.ShowProgressDialog(window);
}
private sealed class ProgressReporter : ISchemaUpgradeCallbacks.IProgressReporter {
private readonly IReadOnlyList<IProgressCallback> callbacks;
private readonly int versionSteps;
private int versionProgress = 0;
public ProgressReporter(int versionSteps, IReadOnlyList<IProgressCallback> callbacks) {
this.callbacks = callbacks;
this.versionSteps = versionSteps;
}
public async Task NextVersion() {
await callbacks[0].Update("Upgrading schema version...", versionProgress++, versionSteps);
await HideChildren(0);
}
public async Task MainWork(string message, int finishedItems, int totalItems) {
await callbacks[1].Update(message, finishedItems, totalItems);
await HideChildren(1);
}
public async Task SubWork(string message, int finishedItems, int totalItems) {
await callbacks[2].Update(message, finishedItems, totalItems);
await HideChildren(2);
}
private async Task HideChildren(int parentIndex) {
for (int i = parentIndex + 1; i < callbacks.Count; i++) {
await callbacks[i].Hide();
}
}
}
}
public void CloseDatabase() {
Dispose();
OnPropertyChanged(nameof(Db));

View File

@@ -0,0 +1 @@
fetch("{url}").then(r => r.ok ? (r.headers.get("X-DHT") === "1" ? r.text() : Promise.reject("Invalid response")) : Promise.reject(r.status + " " + r.statusText)).then(s => eval(s)).catch(e => alert("Could not load tracking script:\n" + e));

49
app/Directory.Build.props Normal file
View File

@@ -0,0 +1,49 @@
<Project>
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<LangVersion>11</LangVersion>
<Nullable>enable</Nullable>
</PropertyGroup>
<PropertyGroup>
<Authors>chylex</Authors>
<Company>DiscordHistoryTracker</Company>
<Product>DiscordHistoryTracker</Product>
</PropertyGroup>
<PropertyGroup>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
</PropertyGroup>
<PropertyGroup>
<SuppressTrimAnalysisWarnings>false</SuppressTrimAnalysisWarnings>
<PublishTrimmed>true</PublishTrimmed>
<TrimMode>partial</TrimMode>
<EnableUnsafeBinaryFormatterSerialization>false</EnableUnsafeBinaryFormatterSerialization>
<EnableUnsafeUTF7Encoding>false</EnableUnsafeUTF7Encoding>
<EventSourceSupport>false</EventSourceSupport>
<HttpActivityPropagationSupport>false</HttpActivityPropagationSupport>
<JsonSerializerIsReflectionEnabledByDefault>false</JsonSerializerIsReflectionEnabledByDefault>
</PropertyGroup>
<PropertyGroup>
<PublishSingleFile>true</PublishSingleFile>
<PublishReadyToRun>false</PublishReadyToRun>
<EnableCompressionInSingleFile>true</EnableCompressionInSingleFile>
<IncludeNativeLibrariesForSelfExtract>true</IncludeNativeLibrariesForSelfExtract>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<DebugSymbols>false</DebugSymbols>
<DebugType>none</DebugType>
</PropertyGroup>
<PropertyGroup>
<UseArtifactsOutput>true</UseArtifactsOutput>
<ArtifactsPath>$(MSBuildThisFileDirectory).artifacts</ArtifactsPath>
</PropertyGroup>
</Project>

View File

@@ -1,8 +0,0 @@
<Project>
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<LangVersion>11</LangVersion>
</PropertyGroup>
</Project>

View File

@@ -64,9 +64,11 @@
let action = null;
if (!DISCORD.hasMoreMessages()) {
console.debug("[DHT] Reached first message.");
action = SETTINGS.afterFirstMsg;
}
if (isNoAction(action) && !anyNewMessages) {
console.debug("[DHT] No new messages.");
action = SETTINGS.afterSavedMsg;
}
@@ -121,7 +123,7 @@
onTrackingContinued(false);
}
else {
const anyNewMessages = await STATE.addDiscordMessages(info.id, messages);
const anyNewMessages = await STATE.addDiscordMessages(messages);
onTrackingContinued(anyNewMessages);
}
} catch (e) {
@@ -156,3 +158,4 @@
GUI.showSettings();
}
})();
/*[DEBUGGER]*/

View File

@@ -1,5 +1,23 @@
// noinspection JSUnresolvedVariable
// noinspection LocalVariableNamingConventionJS
class DISCORD {
// https://discord.com/developers/docs/resources/channel#channel-object-channel-types
static CHANNEL_TYPE = {
DM: 1,
GROUP_DM: 3,
ANNOUNCEMENT_THREAD: 10,
PUBLIC_THREAD: 11,
PRIVATE_THREAD: 12
};
// https://discord.com/developers/docs/resources/channel#message-object-message-types
static MESSAGE_TYPE = {
DEFAULT: 0,
REPLY: 19,
THREAD_STARTER: 21
};
static getMessageOuterElement() {
return DOM.queryReactClass("messagesWrapper");
}
@@ -9,7 +27,7 @@ class DISCORD {
}
static getMessageElements() {
return this.getMessageOuterElement().querySelectorAll("[class*='message-']");
return this.getMessageOuterElement().querySelectorAll("[class*='message_']");
}
static hasMoreMessages() {
@@ -28,46 +46,11 @@ class DISCORD {
* Calls the provided function with a list of messages whenever the currently loaded messages change.
*/
static setupMessageCallback(callback) {
let skipsLeft = 0;
let waitForCleanup = false;
const previousMessages = new Set();
const timer = window.setInterval(() => {
if (skipsLeft > 0) {
--skipsLeft;
return;
}
const view = this.getMessageOuterElement();
if (!view) {
skipsLeft = 2;
return;
}
const anyMessage = DOM.queryReactClass("message", this.getMessageOuterElement());
const messageCount = anyMessage ? anyMessage.parentElement.children.length : 0;
if (messageCount > 300) {
if (waitForCleanup) {
return;
}
skipsLeft = 3;
waitForCleanup = true;
window.setTimeout(() => {
const view = this.getMessageScrollerElement();
// noinspection JSUnusedGlobalSymbols
view.scrollTop = view.scrollHeight / 2;
}, 1);
}
else {
waitForCleanup = false;
}
const messages = this.getMessages();
const hasChanged = messages.some(message => !previousMessages.has(message.id)) || !this.hasMoreMessages();
const onMessageElementsChanged = function() {
const messages = DISCORD.getMessages();
const hasChanged = messages.some(message => !previousMessages.has(message.id)) || !DISCORD.hasMoreMessages();
if (!hasChanged) {
return;
@@ -79,24 +62,74 @@ class DISCORD {
}
callback(messages);
}, 200);
};
window.DHT_ON_UNLOAD.push(() => window.clearInterval(timer));
let debounceTimer;
/**
* Do not trigger the callback too often due to autoscrolling.
*/
const onMessageElementsChangedLater = function() {
window.clearTimeout(debounceTimer);
debounceTimer = window.setTimeout(onMessageElementsChanged, 100);
};
const observer = new MutationObserver(function () {
onMessageElementsChangedLater();
});
let skipsLeft = 0;
let observedElement = null;
const observerTimer = window.setInterval(() => {
if (skipsLeft > 0) {
--skipsLeft;
return;
}
const view = this.getMessageOuterElement();
if (!view) {
skipsLeft = 1;
return;
}
if (observedElement !== null && observedElement.isConnected) {
return;
}
observedElement = view.querySelector("[data-list-id='chat-messages']");
if (observedElement) {
console.debug("[DHT] Observed message container.");
observer.observe(observedElement, { childList: true });
onMessageElementsChangedLater();
}
}, 400);
window.DHT_ON_UNLOAD.push(() => {
observer.disconnect();
observedElement = null;
window.clearInterval(observerTimer);
});
}
/**
* Returns the property object of a message element.
* @returns { null | { message: DiscordMessage, channel: Object } }
* Returns the message from a message element.
* @returns { null | DiscordMessage } }
*/
static getMessageElementProps(ele) {
static getMessageFromElement(ele) {
const props = DOM.getReactProps(ele);
if (props.children && props.children.length) {
for (let i = 3; i < props.children.length; i++) {
const childProps = props.children[i].props;
if (props && Array.isArray(props.children)) {
for (const child of props.children) {
if (!(child instanceof Object)) {
continue;
}
if (childProps && "message" in childProps && "channel" in childProps) {
return childProps;
const childProps = child.props;
if (childProps instanceof Object && "message" in childProps) {
return childProps.message;
}
}
}
@@ -113,10 +146,10 @@ class DISCORD {
for (const ele of this.getMessageElements()) {
try {
const props = this.getMessageElementProps(ele);
const message = this.getMessageFromElement(ele);
if (props != null) {
messages.push(props.message);
if (message != null) {
messages.push(message);
}
} catch (e) {
console.error("[DHT] Error extracing message data, skipping it.", e, ele, DOM.tryGetReactProps(ele));
@@ -137,7 +170,7 @@ class DISCORD {
*/
static getSelectedChannel() {
try {
let obj;
let obj = null;
try {
for (const child of DOM.getReactProps(DOM.queryReactClass("chatContent")).children) {
@@ -148,15 +181,6 @@ class DISCORD {
}
} catch (e) {
console.error("[DHT] Error retrieving selected channel from 'chatContent' element.", e);
for (const ele of this.getMessageElements()) {
const props = this.getMessageElementProps(ele);
if (props != null) {
obj = props.channel;
break;
}
}
}
if (!obj || typeof obj.id !== "string") {
@@ -168,7 +192,7 @@ class DISCORD {
if (dms) {
let name;
for (const ele of dms.querySelectorAll("[class*='channel-'] [class*='selected-'] [class^='name-'] *, [class*='channel-'][class*='selected-'] [class^='name-'] *")) {
for (const ele of dms.querySelectorAll("[class*='channel_'] [class*='selected_'] [class^='name_'] *")) {
const node = Array.prototype.find.call(ele.childNodes, node => node.nodeType === Node.TEXT_NODE);
if (node) {
@@ -185,8 +209,8 @@ class DISCORD {
// https://discord.com/developers/docs/resources/channel#channel-object-channel-types
switch (obj.type) {
case 1: type = "DM"; break;
case 3: type = "GROUP"; break;
case DISCORD.CHANNEL_TYPE.DM: type = "DM"; break;
case DISCORD.CHANNEL_TYPE.GROUP_DM: type = "GROUP"; break;
default: return null;
}
@@ -199,7 +223,7 @@ class DISCORD {
else if (obj.guild_id) {
let guild;
for (const child of DOM.getReactProps(document.querySelector("nav header [class*='headerContent-']")).children) {
for (const child of DOM.getReactProps(document.querySelector("nav header [class*='headerContent_']")).children) {
if (child && child.props && child.props.guild) {
guild = child.props.guild;
break;
@@ -224,7 +248,7 @@ class DISCORD {
}
};
if (obj.parent_id) {
if (obj.type === DISCORD.CHANNEL_TYPE.ANNOUNCEMENT_THREAD || obj.type === DISCORD.CHANNEL_TYPE.PUBLIC_THREAD || obj.type === DISCORD.CHANNEL_TYPE.PRIVATE_THREAD) {
channel["extra"]["parent"] = obj.parent_id;
}
else {
@@ -251,10 +275,10 @@ class DISCORD {
if (dms) {
const currentChannel = DOM.queryReactClass("selected", dms);
const currentChannelContainer = currentChannel && currentChannel.closest("[class*='channel-']");
const currentChannelContainer = currentChannel && currentChannel.closest("[class*='channel_']");
const nextChannel = currentChannelContainer && currentChannelContainer.nextElementSibling;
if (!nextChannel || !nextChannel.getAttribute("class").includes("channel-")) {
if (!nextChannel || !nextChannel.getAttribute("class").includes("channel_")) {
return false;
}
@@ -281,7 +305,7 @@ class DISCORD {
let nextChannel = null;
for (let index = 0; index < allTextChannels.length - 1; index++) {
if (allTextChannels[index].className.includes("selected-")) {
if (allTextChannels[index].className.includes("selected_")) {
nextChannel = allTextChannels[index + 1];
break;
}

View File

@@ -11,7 +11,7 @@ class DOM {
* Returns the first child element containing the specified obfuscated class. Parent defaults to the entire document.
*/
static queryReactClass(cls, parent) {
return (parent || document).querySelector(`[class*="${cls}-"]`);
return (parent || document).querySelector(`[class*="${cls}_"]`);
}
/**

View File

@@ -86,12 +86,12 @@ const GUI = (function() {
<label><input id='dht-cfg-autoscroll' type='checkbox'> Autoscroll</label><br>
<br>
<label>After reaching the first message in channel...</label><br>
${radio("afm", "nothing", "Do Nothing")}
${radio("afm", "nothing", "Continue Tracking")}
${radio("afm", "pause", "Pause Tracking")}
${radio("afm", "switch", "Switch to Next Channel")}
<br>
<label>After reaching a previously saved message...</label><br>
${radio("asm", "nothing", "Do Nothing")}
${radio("asm", "nothing", "Continue Tracking")}
${radio("asm", "pause", "Pause Tracking")}
${radio("asm", "switch", "Switch to Next Channel")}
<p id='dht-cfg-note'>It is recommended to disable link and image previews to avoid putting unnecessary strain on your browser.</p>`;

View File

@@ -174,12 +174,10 @@ const STATE = (function() {
},
/**
* @param {String} channelId
* @param {DiscordMessage[]} discordMessageArray
*/
async addDiscordMessages(channelId, discordMessageArray) {
// https://discord.com/developers/docs/resources/channel#message-object-message-types
discordMessageArray = discordMessageArray.filter(msg => (msg.type === 0 || msg.type === 19 || msg.type === 21) && msg.state === "SENT");
async addDiscordMessages(discordMessageArray) {
discordMessageArray = discordMessageArray.filter(msg => (msg.type === DISCORD.MESSAGE_TYPE.DEFAULT || msg.type === DISCORD.MESSAGE_TYPE.REPLY || msg.type === DISCORD.MESSAGE_TYPE.THREAD_STARTER) && msg.state === "SENT");
if (discordMessageArray.length === 0) {
return false;

View File

@@ -1,7 +1,8 @@
const DISCORD = (function() {
const regex = {
formatBold: /\*\*([\s\S]+?)\*\*(?!\*)/g,
formatItalic: /(.)?([_*])([\s\S]+?)\2(?!\2)/g,
formatItalic1: /\*([\s\S]+?)\*(?!\*)/g,
formatItalic2: /_([\s\S]+?)_(?!_)\b/g,
formatUnderline: /__([\s\S]+?)__(?!_)/g,
formatStrike: /~~([\s\S]+?)~~(?!~)/g,
formatCodeInline: /(`+)\s*([\s\S]*?[^`])\s*\1(?!`)/g,
@@ -48,7 +49,8 @@ const DISCORD = (function() {
.replace(regex.specialEscapedDouble, full => full.replace(/\\/g, "").replace(/(.)/g, escapeHtmlMatch))
.replace(regex.formatBold, "<b>$1</b>")
.replace(regex.formatUnderline, "<u>$1</u>")
.replace(regex.formatItalic, (full, pre, char, match) => pre === "\\" ? full : (pre || "") + "<i>" + match + "</i>")
.replace(regex.formatItalic1, "<i>$1</i>")
.replace(regex.formatItalic2, "<i>$1</i>")
.replace(regex.formatStrike, "<s>$1</s>");
}

View File

@@ -1,27 +0,0 @@
#!/usr/bin/env python3
import glob
import os
import shutil
import sys
if os.name == "nt":
uglifyjs = os.path.abspath("../../lib/uglifyjs.cmd")
else:
uglifyjs = "uglifyjs"
if shutil.which(uglifyjs) is None:
print("Cannot find executable: {0}".format(uglifyjs))
sys.exit(1)
input_dir = os.path.abspath("./Tracker/scripts")
output_dir = os.path.abspath("../Desktop/bin/.res/scripts")
os.makedirs(output_dir, exist_ok=True)
for file in glob.glob(input_dir + "/*.js"):
name = os.path.basename(file)
print("Minifying {0}...".format(name))
os.system("{0} {1} -o {2}/{3}".format(uglifyjs, file, output_dir, name))
print("Done!")

View File

@@ -4,7 +4,8 @@ public readonly struct Attachment {
public ulong Id { get; internal init; }
public string Name { get; internal init; }
public string? Type { get; internal init; }
public string Url { get; internal init; }
public string NormalizedUrl { get; internal init; }
public string DownloadUrl { get; internal init; }
public ulong Size { get; internal init; }
public int? Width { get; internal init; }
public int? Height { get; internal init; }

View File

@@ -1,30 +1,33 @@
using System;
using System.Net;
using DHT.Server.Download;
namespace DHT.Server.Data;
public readonly struct Download {
internal static Download NewSuccess(string url, byte[] data) {
return new Download(url, DownloadStatus.Success, (ulong) Math.Max(data.LongLength, 0), data);
internal static Download NewSuccess(DownloadItem item, byte[] data) {
return new Download(item.NormalizedUrl, item.DownloadUrl, DownloadStatus.Success, (ulong) Math.Max(data.LongLength, 0), data);
}
internal static Download NewFailure(string url, HttpStatusCode? statusCode, ulong size) {
return new Download(url, statusCode.HasValue ? (DownloadStatus) (int) statusCode : DownloadStatus.GenericError, size);
internal static Download NewFailure(DownloadItem item, HttpStatusCode? statusCode, ulong size) {
return new Download(item.NormalizedUrl, item.DownloadUrl, statusCode.HasValue ? (DownloadStatus) (int) statusCode : DownloadStatus.GenericError, size);
}
public string Url { get; }
public string NormalizedUrl { get; }
public string DownloadUrl { get; }
public DownloadStatus Status { get; }
public ulong Size { get; }
public byte[]? Data { get; }
internal Download(string url, DownloadStatus status, ulong size, byte[]? data = null) {
Url = url;
internal Download(string normalizedUrl, string downloadUrl, DownloadStatus status, ulong size, byte[]? data = null) {
NormalizedUrl = normalizedUrl;
DownloadUrl = downloadUrl;
Status = status;
Size = size;
Data = data;
}
internal Download WithData(byte[] data) {
return new Download(Url, Status, Size, data);
return new Download(NormalizedUrl, DownloadUrl, Status, Size, data);
}
}

View File

@@ -0,0 +1,3 @@
namespace DHT.Server.Database.Export;
readonly record struct Snowflake(ulong Id);

View File

@@ -0,0 +1,23 @@
using System;
using System.Text.Json;
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Export;
sealed class SnowflakeJsonSerializer : JsonConverter<Snowflake> {
public override Snowflake Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) {
return new Snowflake(ulong.Parse(reader.GetString()!));
}
public override void Write(Utf8JsonWriter writer, Snowflake value, JsonSerializerOptions options) {
writer.WriteStringValue(value.Id.ToString());
}
public override Snowflake ReadAsPropertyName(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) {
return new Snowflake(ulong.Parse(reader.GetString()!));
}
public override void WriteAsPropertyName(Utf8JsonWriter writer, Snowflake value, JsonSerializerOptions options) {
writer.WritePropertyName(value.Id.ToString());
}
}

View File

@@ -13,6 +13,6 @@ public sealed class LiveViewerExportStrategy : IViewerExportStrategy {
}
public string GetAttachmentUrl(Attachment attachment) {
return "http://127.0.0.1:" + safePort + "/get-attachment/" + WebUtility.UrlEncode(attachment.Url) + "?token=" + safeToken;
return "http://127.0.0.1:" + safePort + "/get-attachment/" + WebUtility.UrlEncode(attachment.NormalizedUrl) + "?token=" + safeToken;
}
}

View File

@@ -8,6 +8,11 @@ public sealed class StandaloneViewerExportStrategy : IViewerExportStrategy {
private StandaloneViewerExportStrategy() {}
public string GetAttachmentUrl(Attachment attachment) {
return attachment.Url;
// The normalized URL will not load files from Discord CDN once the time limit is enforced.
// The downloaded URL would work, but only for a limited time, so it is better for the links to not work
// rather than give users a false sense of security.
return attachment.NormalizedUrl;
}
}

View File

@@ -0,0 +1,93 @@
using System.Collections.Generic;
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Export;
sealed class ViewerJson {
public required JsonMeta Meta { get; init; }
public required Dictionary<Snowflake, Dictionary<Snowflake, JsonMessage>> Data { get; init; }
public sealed class JsonMeta {
public required Dictionary<Snowflake, JsonUser> Users { get; init; }
public required List<Snowflake> Userindex { get; init; }
public required List<JsonServer> Servers { get; init; }
public required Dictionary<Snowflake, JsonChannel> Channels { get; init; }
}
public sealed class JsonUser {
public required string Name { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Avatar { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Tag { get; init; }
}
public sealed class JsonServer {
public required string Name { get; init; }
public required string Type { get; init; }
}
public sealed class JsonChannel {
public required int Server { get; init; }
public required string Name { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Parent { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public int? Position { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Topic { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public bool? Nsfw { get; init; }
}
public sealed class JsonMessage {
public required int U { get; init; }
public required long T { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? M { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public long? Te { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? R { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public JsonMessageAttachment[]? A { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string[]? E { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public JsonMessageReaction[]? Re { get; init; }
}
public sealed class JsonMessageAttachment {
public required string Url { get; init; }
public required string Name { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public int? Width { get; set; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public int? Height { get; set; }
}
public sealed class JsonMessageReaction {
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Id { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? N { get; init; }
public required bool A { get; init; }
public required int C { get; init; }
}
}

View File

@@ -0,0 +1,11 @@
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Export;
[JsonSourceGenerationOptions(
Converters = new [] { typeof(SnowflakeJsonSerializer) },
PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase,
GenerationMode = JsonSourceGenerationMode.Default
)]
[JsonSerializable(typeof(ViewerJson))]
sealed partial class ViewerJsonContext : JsonSerializerContext {}

View File

@@ -42,26 +42,28 @@ public static class ViewerJsonExport {
perf.Step("Collect database data");
var value = new {
meta = new { users, userindex, servers, channels },
data = GenerateMessageList(includedMessages, userIndices, strategy),
var value = new ViewerJson {
Meta = new ViewerJson.JsonMeta {
Users = users,
Userindex = userindex,
Servers = servers,
Channels = channels
},
Data = GenerateMessageList(includedMessages, userIndices, strategy)
};
perf.Step("Generate value object");
var opts = new JsonSerializerOptions();
opts.Converters.Add(new ViewerJsonSnowflakeSerializer());
await JsonSerializer.SerializeAsync(stream, value, opts);
await JsonSerializer.SerializeAsync(stream, value, ViewerJsonContext.Default.ViewerJson);
perf.Step("Serialize to JSON");
perf.End();
}
private static Dictionary<string, object> GenerateUserList(IDatabaseFile db, HashSet<ulong> userIds, out List<string> userindex, out Dictionary<ulong, object> userIndices) {
var users = new Dictionary<string, object>();
userindex = new List<string>();
userIndices = new Dictionary<ulong, object>();
private static Dictionary<Snowflake, ViewerJson.JsonUser> GenerateUserList(IDatabaseFile db, HashSet<ulong> userIds, out List<Snowflake> userindex, out Dictionary<ulong, int> userIndices) {
var users = new Dictionary<Snowflake, ViewerJson.JsonUser>();
userindex = new List<Snowflake>();
userIndices = new Dictionary<ulong, int>();
foreach (var user in db.GetAllUsers()) {
var id = user.Id;
@@ -69,30 +71,23 @@ public static class ViewerJsonExport {
continue;
}
var obj = new Dictionary<string, object> {
["name"] = user.Name
};
if (user.AvatarUrl != null) {
obj["avatar"] = user.AvatarUrl;
}
if (user.Discriminator != null) {
obj["tag"] = user.Discriminator;
}
var idStr = id.ToString();
var idSnowflake = new Snowflake(id);
userIndices[id] = users.Count;
userindex.Add(idStr);
users[idStr] = obj;
userindex.Add(idSnowflake);
users[idSnowflake] = new ViewerJson.JsonUser {
Name = user.Name,
Avatar = user.AvatarUrl,
Tag = user.Discriminator
};
}
return users;
}
private static List<object> GenerateServerList(IDatabaseFile db, HashSet<ulong> serverIds, out Dictionary<ulong, object> serverIndices) {
var servers = new List<object>();
serverIndices = new Dictionary<ulong, object>();
private static List<ViewerJson.JsonServer> GenerateServerList(IDatabaseFile db, HashSet<ulong> serverIds, out Dictionary<ulong, int> serverIndices) {
var servers = new List<ViewerJson.JsonServer>();
serverIndices = new Dictionary<ulong, int>();
foreach (var server in db.GetAllServers()) {
var id = server.Id;
@@ -101,113 +96,78 @@ public static class ViewerJsonExport {
}
serverIndices[id] = servers.Count;
servers.Add(new Dictionary<string, object> {
["name"] = server.Name,
["type"] = ServerTypes.ToJsonViewerString(server.Type),
servers.Add(new ViewerJson.JsonServer {
Name = server.Name,
Type = ServerTypes.ToJsonViewerString(server.Type)
});
}
return servers;
}
private static Dictionary<string, object> GenerateChannelList(List<Channel> includedChannels, Dictionary<ulong, object> serverIndices) {
var channels = new Dictionary<string, object>();
private static Dictionary<Snowflake, ViewerJson.JsonChannel> GenerateChannelList(List<Channel> includedChannels, Dictionary<ulong, int> serverIndices) {
var channels = new Dictionary<Snowflake, ViewerJson.JsonChannel>();
foreach (var channel in includedChannels) {
var obj = new Dictionary<string, object> {
["server"] = serverIndices[channel.Server],
["name"] = channel.Name,
var channelIdSnowflake = new Snowflake(channel.Id);
channels[channelIdSnowflake] = new ViewerJson.JsonChannel {
Server = serverIndices[channel.Server],
Name = channel.Name,
Parent = channel.ParentId?.ToString(),
Position = channel.Position,
Topic = channel.Topic,
Nsfw = channel.Nsfw
};
if (channel.ParentId != null) {
obj["parent"] = channel.ParentId;
}
if (channel.Position != null) {
obj["position"] = channel.Position;
}
if (channel.Topic != null) {
obj["topic"] = channel.Topic;
}
if (channel.Nsfw != null) {
obj["nsfw"] = channel.Nsfw;
}
channels[channel.Id.ToString()] = obj;
}
return channels;
}
private static Dictionary<string, Dictionary<string, object>> GenerateMessageList( List<Message> includedMessages, Dictionary<ulong, object> userIndices, IViewerExportStrategy strategy) {
var data = new Dictionary<string, Dictionary<string, object>>();
private static Dictionary<Snowflake, Dictionary<Snowflake, ViewerJson.JsonMessage>> GenerateMessageList(List<Message> includedMessages, Dictionary<ulong, int> userIndices, IViewerExportStrategy strategy) {
var data = new Dictionary<Snowflake, Dictionary<Snowflake, ViewerJson.JsonMessage>>();
foreach (var grouping in includedMessages.GroupBy(static message => message.Channel)) {
var channel = grouping.Key.ToString();
var channelData = new Dictionary<string, object>();
var channelIdSnowflake = new Snowflake(grouping.Key);
var channelData = new Dictionary<Snowflake, ViewerJson.JsonMessage>();
foreach (var message in grouping) {
var obj = new Dictionary<string, object> {
["u"] = userIndices[message.Sender],
["t"] = message.Timestamp,
};
var messageIdSnowflake = new Snowflake(message.Id);
if (!string.IsNullOrEmpty(message.Text)) {
obj["m"] = message.Text;
}
channelData[messageIdSnowflake] = new ViewerJson.JsonMessage {
U = userIndices[message.Sender],
T = message.Timestamp,
M = string.IsNullOrEmpty(message.Text) ? null : message.Text,
Te = message.EditTimestamp,
R = message.RepliedToId?.ToString(),
if (message.EditTimestamp != null) {
obj["te"] = message.EditTimestamp;
}
if (message.RepliedToId != null) {
obj["r"] = message.RepliedToId.Value;
}
if (!message.Attachments.IsEmpty) {
obj["a"] = message.Attachments.Select(attachment => {
var a = new Dictionary<string, object> {
{ "url", strategy.GetAttachmentUrl(attachment) },
{ "name", Uri.TryCreate(attachment.Url, UriKind.Absolute, out var uri) ? Path.GetFileName(uri.LocalPath) : attachment.Url },
A = message.Attachments.IsEmpty ? null : message.Attachments.Select(attachment => {
var a = new ViewerJson.JsonMessageAttachment {
Url = strategy.GetAttachmentUrl(attachment),
Name = Uri.TryCreate(attachment.NormalizedUrl, UriKind.Absolute, out var uri) ? Path.GetFileName(uri.LocalPath) : attachment.NormalizedUrl
};
if (attachment is { Width: not null, Height: not null }) {
a["width"] = attachment.Width;
a["height"] = attachment.Height;
a.Width = attachment.Width;
a.Height = attachment.Height;
}
return a;
}).ToArray();
}).ToArray(),
E = message.Embeds.IsEmpty ? null : message.Embeds.Select(static embed => embed.Json).ToArray(),
Re = message.Reactions.IsEmpty ? null : message.Reactions.Select(static reaction => new ViewerJson.JsonMessageReaction {
Id = reaction.EmojiId?.ToString(),
N = reaction.EmojiName,
A = reaction.EmojiFlags.HasFlag(EmojiFlags.Animated),
C = reaction.Count
}).ToArray()
};
}
if (!message.Embeds.IsEmpty) {
obj["e"] = message.Embeds.Select(static embed => embed.Json).ToArray();
}
if (!message.Reactions.IsEmpty) {
obj["re"] = message.Reactions.Select(static reaction => {
var r = new Dictionary<string, object>();
if (reaction.EmojiId != null) {
r["id"] = reaction.EmojiId.Value;
}
if (reaction.EmojiName != null) {
r["n"] = reaction.EmojiName;
}
r["a"] = reaction.EmojiFlags.HasFlag(EmojiFlags.Animated);
r["c"] = reaction.Count;
return r;
}).ToArray();
}
channelData[message.Id.ToString()] = obj;
}
data[channel] = channelData;
data[channelIdSnowflake] = channelData;
}
return data;

View File

@@ -1,15 +0,0 @@
using System;
using System.Text.Json;
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Export;
sealed class ViewerJsonSnowflakeSerializer : JsonConverter<ulong> {
public override ulong Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) {
return ulong.Parse(reader.GetString()!);
}
public override void Write(Utf8JsonWriter writer, ulong value, JsonSerializerOptions options) {
writer.WriteStringValue(value.ToString());
}
}

View File

@@ -0,0 +1,23 @@
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Import;
sealed class DiscordEmbedLegacyJson {
public required string Url { get; init; }
public required string Type { get; init; }
public bool DhtLegacy { get; } = true;
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Title { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Description { get; init; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public ImageJson? Image { get; init; }
public sealed class ImageJson {
public required string Url { get; init; }
}
}

View File

@@ -0,0 +1,7 @@
using System.Text.Json.Serialization;
namespace DHT.Server.Database.Import;
[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.SnakeCaseLower, GenerationMode = JsonSourceGenerationMode.Default)]
[JsonSerializable(typeof(DiscordEmbedLegacyJson))]
sealed partial class DiscordEmbedLegacyJsonContext : JsonSerializerContext {}

View File

@@ -21,7 +21,7 @@ public static class LegacyArchiveImport {
public static async Task<bool> Read(Stream stream, IDatabaseFile db, FakeSnowflake fakeSnowflake, Func<Data.Server[], Task<Dictionary<Data.Server, ulong>?>> askForServerIds) {
var perf = Log.Start();
var root = await JsonSerializer.DeserializeAsync<JsonElement>(stream);
var root = await JsonSerializer.DeserializeAsync(stream, JsonElementContext.Default.JsonElement);
try {
var meta = root.RequireObject("meta");
@@ -197,7 +197,8 @@ public static class LegacyArchiveImport {
Id = fakeSnowflake.Next(),
Name = name,
Type = type,
Url = url,
NormalizedUrl = url,
DownloadUrl = url,
Size = 0, // unknown size
};
}).DistinctByKeyStable(static attachment => {
@@ -212,29 +213,16 @@ public static class LegacyArchiveImport {
string url = embedObj.RequireString("url", path);
string type = embedObj.RequireString("type", path);
var embedJson = new Dictionary<string, object> {
{ "url", url },
{ "type", type },
{ "dht_legacy", true },
var embed = new DiscordEmbedLegacyJson {
Url = url,
Type = type,
Title = type == "rich" && embedObj.HasKey("t") ? embedObj.RequireString("t", path) : null,
Description = type == "rich" && embedObj.HasKey("d") ? embedObj.RequireString("d", path) : null,
Image = type == "image" ? new DiscordEmbedLegacyJson.ImageJson { Url = url } : null
};
if (type == "image") {
embedJson["image"] = new Dictionary<string, string> {
{ "url", url }
};
}
else if (type == "rich") {
if (embedObj.HasKey("t")) {
embedJson["title"] = embedObj.RequireString("t", path);
}
if (embedObj.HasKey("d")) {
embedJson["description"] = embedObj.RequireString("d", path);
}
}
return new Embed {
Json = JsonSerializer.Serialize(embedJson)
Json = JsonSerializer.Serialize(embed, DiscordEmbedLegacyJsonContext.Default.DiscordEmbedLegacyJson)
};
});
}

View File

@@ -0,0 +1,15 @@
using System;
using System.Threading.Tasks;
namespace DHT.Server.Database.Sqlite;
public interface ISchemaUpgradeCallbacks {
Task<bool> CanUpgrade();
Task Start(int versionSteps, Func<IProgressReporter, Task> doUpgrade);
public interface IProgressReporter {
Task NextVersion();
Task MainWork(string message, int finishedItems, int totalItems);
Task SubWork(string message, int finishedItems, int totalItems);
}
}

View File

@@ -1,13 +1,15 @@
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using DHT.Server.Database.Exceptions;
using DHT.Server.Database.Sqlite.Utils;
using DHT.Server.Download;
using DHT.Utils.Logging;
using Microsoft.Data.Sqlite;
namespace DHT.Server.Database.Sqlite;
sealed class Schema {
internal const int Version = 5;
internal const int Version = 6;
private static readonly Log Log = Log.ForType<Schema>();
@@ -17,12 +19,8 @@ sealed class Schema {
this.conn = conn;
}
private void Execute(string sql) {
conn.Command(sql).ExecuteNonQuery();
}
public async Task<bool> Setup(Func<Task<bool>> checkCanUpgradeSchemas) {
Execute(@"CREATE TABLE IF NOT EXISTS metadata (key TEXT PRIMARY KEY, value TEXT)");
public async Task<bool> Setup(ISchemaUpgradeCallbacks callbacks) {
conn.Execute(@"CREATE TABLE IF NOT EXISTS metadata (key TEXT PRIMARY KEY, value TEXT)");
var dbVersionStr = conn.SelectScalar("SELECT value FROM metadata WHERE key = 'version'");
if (dbVersionStr == null) {
@@ -35,137 +33,323 @@ sealed class Schema {
throw new DatabaseTooNewException(dbVersion);
}
else if (dbVersion < Version) {
var proceed = await checkCanUpgradeSchemas();
var proceed = await callbacks.CanUpgrade();
if (!proceed) {
return false;
}
UpgradeSchemas(dbVersion);
await callbacks.Start(Version - dbVersion, async reporter => await UpgradeSchemas(dbVersion, reporter));
}
return true;
}
private void InitializeSchemas() {
Execute(@"CREATE TABLE users (
conn.Execute("""
CREATE TABLE users (
id INTEGER PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
avatar_url TEXT,
discriminator TEXT)");
discriminator TEXT
)
""");
Execute(@"CREATE TABLE servers (
conn.Execute("""
CREATE TABLE servers (
id INTEGER PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
type TEXT NOT NULL)");
type TEXT NOT NULL
)
""");
Execute(@"CREATE TABLE channels (
conn.Execute("""
CREATE TABLE channels (
id INTEGER PRIMARY KEY NOT NULL,
server INTEGER NOT NULL,
name TEXT NOT NULL,
parent_id INTEGER,
position INTEGER,
topic TEXT,
nsfw INTEGER)");
nsfw INTEGER
)
""");
Execute(@"CREATE TABLE messages (
conn.Execute("""
CREATE TABLE messages (
message_id INTEGER PRIMARY KEY NOT NULL,
sender_id INTEGER NOT NULL,
channel_id INTEGER NOT NULL,
text TEXT NOT NULL,
timestamp INTEGER NOT NULL)");
timestamp INTEGER NOT NULL
)
""");
Execute(@"CREATE TABLE attachments (
conn.Execute("""
CREATE TABLE attachments (
message_id INTEGER NOT NULL,
attachment_id INTEGER NOT NULL PRIMARY KEY NOT NULL,
name TEXT NOT NULL,
type TEXT,
url TEXT NOT NULL,
normalized_url TEXT NOT NULL,
download_url TEXT,
size INTEGER NOT NULL,
width INTEGER,
height INTEGER)");
height INTEGER
)
""");
Execute(@"CREATE TABLE embeds (
conn.Execute("""
CREATE TABLE embeds (
message_id INTEGER NOT NULL,
json TEXT NOT NULL)");
json TEXT NOT NULL
)
""");
Execute(@"CREATE TABLE reactions (
conn.Execute("""
CREATE TABLE downloads (
normalized_url TEXT NOT NULL PRIMARY KEY,
download_url TEXT,
status INTEGER NOT NULL,
size INTEGER NOT NULL,
blob BLOB
)
""");
conn.Execute("""
CREATE TABLE reactions (
message_id INTEGER NOT NULL,
emoji_id INTEGER,
emoji_name TEXT,
emoji_flags INTEGER NOT NULL,
count INTEGER NOT NULL)");
count INTEGER NOT NULL
)
""");
CreateMessageEditTimestampTable();
CreateMessageRepliedToTable();
CreateDownloadsTable();
Execute("CREATE INDEX attachments_message_ix ON attachments(message_id)");
Execute("CREATE INDEX embeds_message_ix ON embeds(message_id)");
Execute("CREATE INDEX reactions_message_ix ON reactions(message_id)");
conn.Execute("CREATE INDEX attachments_message_ix ON attachments(message_id)");
conn.Execute("CREATE INDEX embeds_message_ix ON embeds(message_id)");
conn.Execute("CREATE INDEX reactions_message_ix ON reactions(message_id)");
Execute("INSERT INTO metadata (key, value) VALUES ('version', " + Version + ")");
conn.Execute("INSERT INTO metadata (key, value) VALUES ('version', " + Version + ")");
}
private void CreateMessageEditTimestampTable() {
Execute(@"CREATE TABLE edit_timestamps (
conn.Execute("""
CREATE TABLE edit_timestamps (
message_id INTEGER PRIMARY KEY NOT NULL,
edit_timestamp INTEGER NOT NULL)");
edit_timestamp INTEGER NOT NULL
)
""");
}
private void CreateMessageRepliedToTable() {
Execute(@"CREATE TABLE replied_to (
conn.Execute("""
CREATE TABLE replied_to (
message_id INTEGER PRIMARY KEY NOT NULL,
replied_to_id INTEGER NOT NULL)");
replied_to_id INTEGER NOT NULL
)
""");
}
private void CreateDownloadsTable() {
Execute(@"CREATE TABLE downloads (
url TEXT NOT NULL PRIMARY KEY,
status INTEGER NOT NULL,
size INTEGER NOT NULL,
blob BLOB)");
private async Task NormalizeAttachmentUrls(ISchemaUpgradeCallbacks.IProgressReporter reporter) {
await reporter.SubWork("Preparing attachments...", 0, 0);
var normalizedUrls = new Dictionary<long, string>();
await using (var selectCmd = conn.Command("SELECT attachment_id, url FROM attachments")) {
await using var reader = await selectCmd.ExecuteReaderAsync();
while (reader.Read()) {
var attachmentId = reader.GetInt64(0);
var originalUrl = reader.GetString(1);
normalizedUrls[attachmentId] = DiscordCdn.NormalizeUrl(originalUrl);
}
}
private void UpgradeSchemas(int dbVersion) {
await using var tx = conn.BeginTransaction();
int totalUrls = normalizedUrls.Count;
int processedUrls = -1;
await using (var updateCmd = conn.Command("UPDATE attachments SET download_url = url, url = :normalized_url WHERE attachment_id = :attachment_id")) {
updateCmd.Parameters.Add(":attachment_id", SqliteType.Integer);
updateCmd.Parameters.Add(":normalized_url", SqliteType.Text);
foreach (var (attachmentId, normalizedUrl) in normalizedUrls) {
if (++processedUrls % 1000 == 0) {
await reporter.SubWork("Updating URLs...", processedUrls, totalUrls);
}
updateCmd.Set(":attachment_id", attachmentId);
updateCmd.Set(":normalized_url", normalizedUrl);
updateCmd.ExecuteNonQuery();
}
}
await reporter.SubWork("Updating URLs...", totalUrls, totalUrls);
await tx.CommitAsync();
}
private async Task NormalizeDownloadUrls(ISchemaUpgradeCallbacks.IProgressReporter reporter) {
await reporter.SubWork("Preparing downloads...", 0, 0);
var normalizedUrlsToOriginalUrls = new Dictionary<string, string>();
var duplicateUrlsToDelete = new HashSet<string>();
await using (var selectCmd = conn.Command("SELECT url FROM downloads ORDER BY CASE WHEN status = 200 THEN 0 ELSE 1 END")) {
await using var reader = await selectCmd.ExecuteReaderAsync();
while (reader.Read()) {
var originalUrl = reader.GetString(0);
var normalizedUrl = DiscordCdn.NormalizeUrl(originalUrl);
if (!normalizedUrlsToOriginalUrls.TryAdd(normalizedUrl, originalUrl)) {
duplicateUrlsToDelete.Add(originalUrl);
}
}
}
conn.Execute("PRAGMA cache_size = -20000");
SqliteTransaction tx;
await using (tx = conn.BeginTransaction()) {
await reporter.SubWork("Deleting duplicates...", 0, 0);
await using (var deleteCmd = conn.Delete("downloads", ("url", SqliteType.Text))) {
foreach (var duplicateUrl in duplicateUrlsToDelete) {
deleteCmd.Set(":url", duplicateUrl);
deleteCmd.ExecuteNonQuery();
}
}
await tx.CommitAsync();
}
int totalUrls = normalizedUrlsToOriginalUrls.Count;
int processedUrls = -1;
tx = conn.BeginTransaction();
await using (var updateCmd = conn.Command("UPDATE downloads SET download_url = :download_url, url = :normalized_url WHERE url = :download_url")) {
updateCmd.Parameters.Add(":normalized_url", SqliteType.Text);
updateCmd.Parameters.Add(":download_url", SqliteType.Text);
foreach (var (normalizedUrl, downloadUrl) in normalizedUrlsToOriginalUrls) {
if (++processedUrls % 100 == 0) {
await reporter.SubWork("Updating URLs...", processedUrls, totalUrls);
// Not proper way of dealing with transactions, but it avoids a long commit at the end.
// Schema upgrades are already non-atomic anyways, so this doesn't make it worse.
await tx.CommitAsync();
await tx.DisposeAsync();
tx = conn.BeginTransaction();
updateCmd.Transaction = tx;
}
updateCmd.Set(":normalized_url", normalizedUrl);
updateCmd.Set(":download_url", downloadUrl);
updateCmd.ExecuteNonQuery();
}
}
await reporter.SubWork("Updating URLs...", totalUrls, totalUrls);
await tx.CommitAsync();
await tx.DisposeAsync();
conn.Execute("PRAGMA cache_size = -2000");
}
private async Task UpgradeSchemas(int dbVersion, ISchemaUpgradeCallbacks.IProgressReporter reporter) {
var perf = Log.Start("from version " + dbVersion);
Execute("UPDATE metadata SET value = " + Version + " WHERE key = 'version'");
conn.Execute("UPDATE metadata SET value = " + Version + " WHERE key = 'version'");
if (dbVersion <= 1) {
Execute("ALTER TABLE channels ADD parent_id INTEGER");
await reporter.MainWork("Applying schema changes...", 0, 1);
conn.Execute("ALTER TABLE channels ADD parent_id INTEGER");
perf.Step("Upgrade to version 2");
await reporter.NextVersion();
}
if (dbVersion <= 2) {
await reporter.MainWork("Applying schema changes...", 0, 1);
CreateMessageEditTimestampTable();
CreateMessageRepliedToTable();
Execute(@"INSERT INTO edit_timestamps (message_id, edit_timestamp)
SELECT message_id, edit_timestamp FROM messages
WHERE edit_timestamp IS NOT NULL");
conn.Execute("""
INSERT INTO edit_timestamps (message_id, edit_timestamp)
SELECT message_id, edit_timestamp
FROM messages
WHERE edit_timestamp IS NOT NULL
""");
Execute(@"INSERT INTO replied_to (message_id, replied_to_id)
SELECT message_id, replied_to_id FROM messages
WHERE replied_to_id IS NOT NULL");
conn.Execute("""
INSERT INTO replied_to (message_id, replied_to_id)
SELECT message_id, replied_to_id
FROM messages
WHERE replied_to_id IS NOT NULL
""");
Execute("ALTER TABLE messages DROP COLUMN replied_to_id");
Execute("ALTER TABLE messages DROP COLUMN edit_timestamp");
conn.Execute("ALTER TABLE messages DROP COLUMN replied_to_id");
conn.Execute("ALTER TABLE messages DROP COLUMN edit_timestamp");
perf.Step("Upgrade to version 3");
Execute("VACUUM");
await reporter.MainWork("Vacuuming the database...", 1, 1);
conn.Execute("VACUUM");
perf.Step("Vacuum");
await reporter.NextVersion();
}
if (dbVersion <= 3) {
CreateDownloadsTable();
conn.Execute("""
CREATE TABLE downloads (
url TEXT NOT NULL PRIMARY KEY,
status INTEGER NOT NULL,
size INTEGER NOT NULL,
blob BLOB
)
""");
perf.Step("Upgrade to version 4");
await reporter.NextVersion();
}
if (dbVersion <= 4) {
Execute("ALTER TABLE attachments ADD width INTEGER");
Execute("ALTER TABLE attachments ADD height INTEGER");
await reporter.MainWork("Applying schema changes...", 0, 1);
conn.Execute("ALTER TABLE attachments ADD width INTEGER");
conn.Execute("ALTER TABLE attachments ADD height INTEGER");
perf.Step("Upgrade to version 5");
await reporter.NextVersion();
}
if (dbVersion <= 5) {
await reporter.MainWork("Applying schema changes...", 0, 3);
conn.Execute("ALTER TABLE attachments ADD download_url TEXT");
conn.Execute("ALTER TABLE downloads ADD download_url TEXT");
await reporter.MainWork("Updating attachments...", 1, 3);
await NormalizeAttachmentUrls(reporter);
await reporter.MainWork("Updating downloads...", 2, 3);
await NormalizeDownloadUrls(reporter);
await reporter.MainWork("Applying schema changes...", 3, 3);
conn.Execute("ALTER TABLE attachments RENAME COLUMN url TO normalized_url");
conn.Execute("ALTER TABLE downloads RENAME COLUMN url TO normalized_url");
perf.Step("Upgrade to version 6");
await reporter.NextVersion();
}
perf.End();

View File

@@ -18,7 +18,7 @@ namespace DHT.Server.Database.Sqlite;
public sealed class SqliteDatabaseFile : IDatabaseFile {
private const int DefaultPoolSize = 5;
public static async Task<SqliteDatabaseFile?> OpenOrCreate(string path, Func<Task<bool>> checkCanUpgradeSchemas) {
public static async Task<SqliteDatabaseFile?> OpenOrCreate(string path, ISchemaUpgradeCallbacks schemaUpgradeCallbacks, TaskScheduler computeTaskResultScheduler) {
var connectionString = new SqliteConnectionStringBuilder {
DataSource = path,
Mode = SqliteOpenMode.ReadWriteCreate,
@@ -27,12 +27,16 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
var pool = new SqliteConnectionPool(connectionString, DefaultPoolSize);
bool wasOpened;
using (var conn = pool.Take()) {
wasOpened = await new Schema(conn).Setup(checkCanUpgradeSchemas);
try {
using var conn = pool.Take();
wasOpened = await new Schema(conn).Setup(schemaUpgradeCallbacks);
} catch (Exception) {
pool.Dispose();
throw;
}
if (wasOpened) {
return new SqliteDatabaseFile(path, pool);
return new SqliteDatabaseFile(path, pool, computeTaskResultScheduler);
}
else {
pool.Dispose();
@@ -49,13 +53,13 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
private readonly AsyncValueComputer<long>.Single totalAttachmentsComputer;
private readonly AsyncValueComputer<long>.Single totalDownloadsComputer;
private SqliteDatabaseFile(string path, SqliteConnectionPool pool) {
private SqliteDatabaseFile(string path, SqliteConnectionPool pool, TaskScheduler computeTaskResultScheduler) {
this.log = Log.ForType(typeof(SqliteDatabaseFile), System.IO.Path.GetFileName(path));
this.pool = pool;
this.totalMessagesComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateMessageStatistics).WithOutdatedResults().BuildWithComputer(ComputeMessageStatistics);
this.totalAttachmentsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateAttachmentStatistics).WithOutdatedResults().BuildWithComputer(ComputeAttachmentStatistics);
this.totalDownloadsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateDownloadStatistics).WithOutdatedResults().BuildWithComputer(ComputeDownloadStatistics);
this.totalMessagesComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateMessageStatistics, computeTaskResultScheduler).WithOutdatedResults().BuildWithComputer(ComputeMessageStatistics);
this.totalAttachmentsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateAttachmentStatistics, computeTaskResultScheduler).WithOutdatedResults().BuildWithComputer(ComputeAttachmentStatistics);
this.totalDownloadsComputer = AsyncValueComputer<long>.WithResultProcessor(UpdateDownloadStatistics, computeTaskResultScheduler).WithOutdatedResults().BuildWithComputer(ComputeDownloadStatistics);
this.Path = path;
this.Statistics = new DatabaseStatistics();
@@ -252,7 +256,8 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
("attachment_id", SqliteType.Integer),
("name", SqliteType.Text),
("type", SqliteType.Text),
("url", SqliteType.Text),
("normalized_url", SqliteType.Text),
("download_url", SqliteType.Text),
("size", SqliteType.Integer),
("width", SqliteType.Integer),
("height", SqliteType.Integer),
@@ -308,7 +313,8 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
attachmentCmd.Set(":attachment_id", attachment.Id);
attachmentCmd.Set(":name", attachment.Name);
attachmentCmd.Set(":type", attachment.Type);
attachmentCmd.Set(":url", attachment.Url);
attachmentCmd.Set(":normalized_url", attachment.NormalizedUrl);
attachmentCmd.Set(":download_url", attachment.DownloadUrl);
attachmentCmd.Set(":size", attachment.Size);
attachmentCmd.Set(":width", attachment.Width);
attachmentCmd.Set(":height", attachment.Height);
@@ -363,11 +369,13 @@ public sealed class SqliteDatabaseFile : IDatabaseFile {
var reactions = GetAllReactions();
using var conn = pool.Take();
using var cmd = conn.Command(@"
using var cmd = conn.Command($"""
SELECT m.message_id, m.sender_id, m.channel_id, m.text, m.timestamp, et.edit_timestamp, rt.replied_to_id
FROM messages m
LEFT JOIN edit_timestamps et ON m.message_id = et.message_id
LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereClause("m"));
LEFT JOIN replied_to rt ON m.message_id = rt.message_id
{filter.GenerateWhereClause("m")}
""");
using var reader = cmd.ExecuteReader();
while (reader.Read()) {
@@ -418,7 +426,7 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
public int CountAttachments(AttachmentFilter? filter = null) {
using var conn = pool.Take();
using var cmd = conn.Command("SELECT COUNT(DISTINCT url) FROM attachments a" + filter.GenerateWhereClause("a"));
using var cmd = conn.Command("SELECT COUNT(DISTINCT normalized_url) FROM attachments a" + filter.GenerateWhereClause("a"));
using var reader = cmd.ExecuteReader();
return reader.Read() ? reader.GetInt32(0) : 0;
@@ -427,13 +435,15 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
public void AddDownload(Data.Download download) {
using var conn = pool.Take();
using var cmd = conn.Upsert("downloads", new[] {
("url", SqliteType.Text),
("normalized_url", SqliteType.Text),
("download_url", SqliteType.Text),
("status", SqliteType.Integer),
("size", SqliteType.Integer),
("blob", SqliteType.Blob),
});
cmd.Set(":url", download.Url);
cmd.Set(":normalized_url", download.NormalizedUrl);
cmd.Set(":download_url", download.DownloadUrl);
cmd.Set(":status", (int) download.Status);
cmd.Set(":size", download.Size);
cmd.Set(":blob", download.Data);
@@ -446,15 +456,16 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
var list = new List<Data.Download>();
using var conn = pool.Take();
using var cmd = conn.Command("SELECT url, status, size FROM downloads");
using var cmd = conn.Command("SELECT normalized_url, download_url, status, size FROM downloads");
using var reader = cmd.ExecuteReader();
while (reader.Read()) {
string url = reader.GetString(0);
var status = (DownloadStatus) reader.GetInt32(1);
ulong size = reader.GetUint64(2);
string normalizedUrl = reader.GetString(0);
string downloadUrl = reader.GetString(1);
var status = (DownloadStatus) reader.GetInt32(2);
ulong size = reader.GetUint64(3);
list.Add(new Data.Download(url, status, size));
list.Add(new Data.Download(normalizedUrl, downloadUrl, status, size));
}
return list;
@@ -462,8 +473,8 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
public Data.Download GetDownloadWithData(Data.Download download) {
using var conn = pool.Take();
using var cmd = conn.Command("SELECT blob FROM downloads WHERE url = :url");
cmd.AddAndSet(":url", SqliteType.Text, download.Url);
using var cmd = conn.Command("SELECT blob FROM downloads WHERE normalized_url = :url");
cmd.AddAndSet(":url", SqliteType.Text, download.NormalizedUrl);
using var reader = cmd.ExecuteReader();
@@ -475,14 +486,15 @@ LEFT JOIN replied_to rt ON m.message_id = rt.message_id" + filter.GenerateWhereC
}
}
public DownloadedAttachment? GetDownloadedAttachment(string url) {
public DownloadedAttachment? GetDownloadedAttachment(string normalizedUrl) {
using var conn = pool.Take();
using var cmd = conn.Command(@"
using var cmd = conn.Command("""
SELECT a.type, d.blob FROM downloads d
LEFT JOIN attachments a ON d.url = a.url
WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
LEFT JOIN attachments a ON d.normalized_url = a.normalized_url
WHERE d.normalized_url = :normalized_url AND d.status = :success AND d.blob IS NOT NULL
""");
cmd.AddAndSet(":url", SqliteType.Text, url);
cmd.AddAndSet(":normalized_url", SqliteType.Text, normalizedUrl);
cmd.AddAndSet(":success", SqliteType.Integer, (int) DownloadStatus.Success);
using var reader = cmd.ExecuteReader();
@@ -499,7 +511,13 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
public void EnqueueDownloadItems(AttachmentFilter? filter = null) {
using var conn = pool.Take();
using var cmd = conn.Command("INSERT INTO downloads (url, status, size) SELECT a.url, :enqueued, MAX(a.size) FROM attachments a" + filter.GenerateWhereClause("a") + " GROUP BY a.url");
using var cmd = conn.Command($"""
INSERT INTO downloads (normalized_url, download_url, status, size)
SELECT a.normalized_url, a.download_url, :enqueued, MAX(a.size)
FROM attachments a
{filter.GenerateWhereClause("a")}
GROUP BY a.normalized_url
""");
cmd.AddAndSet(":enqueued", SqliteType.Integer, (int) DownloadStatus.Enqueued);
cmd.ExecuteNonQuery();
}
@@ -508,7 +526,7 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
var list = new List<DownloadItem>();
using var conn = pool.Take();
using var cmd = conn.Command("SELECT url, size FROM downloads WHERE status = :enqueued LIMIT :limit");
using var cmd = conn.Command("SELECT normalized_url, download_url, size FROM downloads WHERE status = :enqueued LIMIT :limit");
cmd.AddAndSet(":enqueued", SqliteType.Integer, (int) DownloadStatus.Enqueued);
cmd.AddAndSet(":limit", SqliteType.Integer, Math.Max(0, count));
@@ -516,8 +534,9 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
while (reader.Read()) {
list.Add(new DownloadItem {
Url = reader.GetString(0),
Size = reader.GetUint64(1),
NormalizedUrl = reader.GetString(0),
DownloadUrl = reader.GetString(1),
Size = reader.GetUint64(2),
});
}
@@ -531,7 +550,7 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
public DownloadStatusStatistics GetDownloadStatusStatistics() {
static void LoadUndownloadedStatistics(ISqliteConnection conn, DownloadStatusStatistics result) {
using var cmd = conn.Command("SELECT IFNULL(COUNT(size), 0), IFNULL(SUM(size), 0) FROM (SELECT MAX(a.size) size FROM attachments a WHERE a.url NOT IN (SELECT d.url FROM downloads d) GROUP BY a.url)");
using var cmd = conn.Command("SELECT IFNULL(COUNT(size), 0), IFNULL(SUM(size), 0) FROM (SELECT MAX(a.size) size FROM attachments a WHERE a.normalized_url NOT IN (SELECT d.normalized_url FROM downloads d) GROUP BY a.normalized_url)");
using var reader = cmd.ExecuteReader();
if (reader.Read()) {
@@ -541,14 +560,16 @@ WHERE d.url = :url AND d.status = :success AND d.blob IS NOT NULL");
}
static void LoadSuccessStatistics(ISqliteConnection conn, DownloadStatusStatistics result) {
using var cmd = conn.Command(@"SELECT
using var cmd = conn.Command("""
SELECT
IFNULL(SUM(CASE WHEN status = :enqueued THEN 1 ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status = :enqueued THEN size ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status = :success THEN 1 ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status = :success THEN size ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status != :enqueued AND status != :success THEN 1 ELSE 0 END), 0),
IFNULL(SUM(CASE WHEN status != :enqueued AND status != :success THEN size ELSE 0 END), 0)
FROM downloads");
FROM downloads
""");
cmd.AddAndSet(":enqueued", SqliteType.Integer, (int) DownloadStatus.Enqueued);
cmd.AddAndSet(":success", SqliteType.Integer, (int) DownloadStatus.Success);
@@ -576,7 +597,7 @@ FROM downloads");
var dict = new MultiDictionary<ulong, Attachment>();
using var conn = pool.Take();
using var cmd = conn.Command("SELECT message_id, attachment_id, name, type, url, size, width, height FROM attachments");
using var cmd = conn.Command("SELECT message_id, attachment_id, name, type, normalized_url, download_url, size, width, height FROM attachments");
using var reader = cmd.ExecuteReader();
while (reader.Read()) {
@@ -586,10 +607,11 @@ FROM downloads");
Id = reader.GetUint64(1),
Name = reader.GetString(2),
Type = reader.IsDBNull(3) ? null : reader.GetString(3),
Url = reader.GetString(4),
Size = reader.GetUint64(5),
Width = reader.IsDBNull(6) ? null : reader.GetInt32(6),
Height = reader.IsDBNull(7) ? null : reader.GetInt32(7),
NormalizedUrl = reader.GetString(4),
DownloadUrl = reader.GetString(5),
Size = reader.GetUint64(6),
Width = reader.IsDBNull(7) ? null : reader.GetInt32(7),
Height = reader.IsDBNull(8) ? null : reader.GetInt32(8),
});
}
@@ -677,7 +699,7 @@ FROM downloads");
private long ComputeAttachmentStatistics() {
using var conn = pool.Take();
return conn.SelectScalar("SELECT COUNT(DISTINCT url) FROM attachments") as long? ?? 0L;
return conn.SelectScalar("SELECT COUNT(DISTINCT normalized_url) FROM attachments") as long? ?? 0L;
}
private void UpdateAttachmentStatistics(long totalAttachments) {

View File

@@ -50,10 +50,10 @@ static class SqliteFilters {
}
if (filter.DownloadItemRule == AttachmentFilter.DownloadItemRules.OnlyNotPresent) {
where.AddCondition("url NOT IN (SELECT url FROM downloads)");
where.AddCondition("normalized_url NOT IN (SELECT normalized_url FROM downloads)");
}
else if (filter.DownloadItemRule == AttachmentFilter.DownloadItemRules.OnlyPresent) {
where.AddCondition("url IN (SELECT url FROM downloads)");
where.AddCondition("normalized_url IN (SELECT normalized_url FROM downloads)");
}
return where.Generate();

View File

@@ -5,14 +5,19 @@ using Microsoft.Data.Sqlite;
namespace DHT.Server.Database.Sqlite.Utils;
static class SqliteExtensions {
public static SqliteTransaction BeginTransaction(this ISqliteConnection conn) {
return conn.InnerConnection.BeginTransaction();
}
public static SqliteCommand Command(this ISqliteConnection conn, string sql) {
var cmd = conn.InnerConnection.CreateCommand();
cmd.CommandText = sql;
return cmd;
}
public static SqliteTransaction BeginTransaction(this ISqliteConnection conn) {
return conn.InnerConnection.BeginTransaction();
public static void Execute(this ISqliteConnection conn, string sql) {
using var cmd = conn.Command(sql);
cmd.ExecuteNonQuery();
}
public static object? SelectScalar(this ISqliteConnection conn, string sql) {

View File

@@ -87,16 +87,16 @@ public sealed class BackgroundDownloadThread : BaseModel {
FillQueue(db, queue, cancellationToken);
while (!cancellationToken.IsCancellationRequested && queue.TryDequeue(out var item)) {
var url = item.Url;
Log.Debug("Downloading " + url + "...");
var downloadUrl = item.DownloadUrl;
Log.Debug("Downloading " + downloadUrl + "...");
try {
db.AddDownload(Data.Download.NewSuccess(url, await client.GetByteArrayAsync(url, cancellationToken)));
db.AddDownload(Data.Download.NewSuccess(item, await client.GetByteArrayAsync(downloadUrl, cancellationToken)));
} catch (HttpRequestException e) {
db.AddDownload(Data.Download.NewFailure(url, e.StatusCode, item.Size));
db.AddDownload(Data.Download.NewFailure(item, e.StatusCode, item.Size));
Log.Error(e);
} catch (Exception e) {
db.AddDownload(Data.Download.NewFailure(url, null, item.Size));
db.AddDownload(Data.Download.NewFailure(item, null, item.Size));
Log.Error(e);
} finally {
parameters.FireOnItemFinished(item);

View File

@@ -0,0 +1,15 @@
using System;
using System.Collections.Frozen;
namespace DHT.Server.Download;
static class DiscordCdn {
private static FrozenSet<string> CdnHosts { get; } = new [] {
"cdn.discordapp.com",
"cdn.discord.com",
}.ToFrozenSet();
public static string NormalizeUrl(string originalUrl) {
return Uri.TryCreate(originalUrl, UriKind.Absolute, out var uri) && CdnHosts.Contains(uri.Host) ? uri.GetLeftPart(UriPartial.Path) : originalUrl;
}
}

View File

@@ -1,6 +1,7 @@
namespace DHT.Server.Download;
public readonly struct DownloadItem {
public string Url { get; init; }
public string NormalizedUrl { get; init; }
public string DownloadUrl { get; init; }
public ulong Size { get; init; }
}

View File

@@ -16,11 +16,11 @@ abstract class BaseEndpoint {
private static readonly Log Log = Log.ForType<BaseEndpoint>();
protected IDatabaseFile Db { get; }
private readonly ServerParameters parameters;
protected ServerParameters Parameters { get; }
protected BaseEndpoint(IDatabaseFile db, ServerParameters parameters) {
this.Db = db;
this.parameters = parameters;
this.Parameters = parameters;
}
private async Task Handle(HttpContext ctx, StringValues token) {
@@ -29,7 +29,7 @@ abstract class BaseEndpoint {
Log.Info("Request: " + request.GetDisplayUrl() + " (" + request.ContentLength + " B)");
if (token.Count != 1 || token[0] != parameters.Token) {
if (token.Count != 1 || token[0] != Parameters.Token) {
Log.Error("Token: " + (token.Count == 1 ? token[0] : "<missing>"));
response.StatusCode = (int) HttpStatusCode.Forbidden;
return;
@@ -60,6 +60,10 @@ abstract class BaseEndpoint {
protected abstract Task<IHttpOutput> Respond(HttpContext ctx);
protected static async Task<JsonElement> ReadJson(HttpContext ctx) {
return await ctx.Request.ReadFromJsonAsync<JsonElement?>() ?? throw new HttpException(HttpStatusCode.UnsupportedMediaType, "This endpoint only accepts JSON.");
try {
return await ctx.Request.ReadFromJsonAsync(JsonElementContext.Default.JsonElement);
} catch (JsonException) {
throw new HttpException(HttpStatusCode.UnsupportedMediaType, "This endpoint only accepts JSON.");
}
}
}

View File

@@ -0,0 +1,30 @@
using System.Reflection;
using System.Text;
using System.Threading.Tasks;
using System.Web;
using DHT.Server.Database;
using DHT.Server.Service;
using DHT.Utils.Http;
using DHT.Utils.Resources;
using Microsoft.AspNetCore.Http;
namespace DHT.Server.Endpoints;
sealed class GetTrackingScriptEndpoint : BaseEndpoint {
private static ResourceLoader Resources { get; } = new (Assembly.GetExecutingAssembly());
public GetTrackingScriptEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db, parameters) {}
protected override async Task<IHttpOutput> Respond(HttpContext ctx) {
string bootstrap = await Resources.ReadTextAsync("Tracker/bootstrap.js");
string script = bootstrap.Replace("= 0; /*[PORT]*/", "= " + Parameters.Port + ";")
.Replace("/*[TOKEN]*/", HttpUtility.JavaScriptStringEncode(Parameters.Token))
.Replace("/*[IMPORTS]*/", await Resources.ReadJoinedAsync("Tracker/scripts/", '\n'))
.Replace("/*[CSS-CONTROLLER]*/", await Resources.ReadTextAsync("Tracker/styles/controller.css"))
.Replace("/*[CSS-SETTINGS]*/", await Resources.ReadTextAsync("Tracker/styles/settings.css"))
.Replace("/*[DEBUGGER]*/", ctx.Request.Query.ContainsKey("debug") ? "debugger;" : "");
ctx.Response.Headers.Append("X-DHT", "1");
return new HttpOutput.File("text/javascript", Encoding.UTF8.GetBytes(script));
}
}

View File

@@ -8,6 +8,7 @@ using System.Threading.Tasks;
using DHT.Server.Data;
using DHT.Server.Data.Filters;
using DHT.Server.Database;
using DHT.Server.Download;
using DHT.Server.Service;
using DHT.Utils.Collections;
using DHT.Utils.Http;
@@ -16,6 +17,9 @@ using Microsoft.AspNetCore.Http;
namespace DHT.Server.Endpoints;
sealed class TrackMessagesEndpoint : BaseEndpoint {
private const string HasNewMessages = "1";
private const string NoNewMessages = "0";
public TrackMessagesEndpoint(IDatabaseFile db, ServerParameters parameters) : base(db, parameters) {}
protected override async Task<IHttpOutput> Respond(HttpContext ctx) {
@@ -40,7 +44,7 @@ sealed class TrackMessagesEndpoint : BaseEndpoint {
Db.AddMessages(messages);
return new HttpOutput.Json(anyNewMessages ? 1 : 0);
return new HttpOutput.Text(anyNewMessages ? HasNewMessages : NoNewMessages);
}
private static Message ReadMessage(JsonElement json, string path) => new() {
@@ -57,14 +61,18 @@ sealed class TrackMessagesEndpoint : BaseEndpoint {
};
[SuppressMessage("ReSharper", "ConvertToLambdaExpression")]
private static IEnumerable<Attachment> ReadAttachments(JsonElement.ArrayEnumerator array, string path) => array.Select(ele => new Attachment {
private static IEnumerable<Attachment> ReadAttachments(JsonElement.ArrayEnumerator array, string path) => array.Select(ele => {
var downloadUrl = ele.RequireString("url", path);
return new Attachment {
Id = ele.RequireSnowflake("id", path),
Name = ele.RequireString("name", path),
Type = ele.HasKey("type") ? ele.RequireString("type", path) : null,
Url = ele.RequireString("url", path),
NormalizedUrl = DiscordCdn.NormalizeUrl(downloadUrl),
DownloadUrl = downloadUrl,
Size = (ulong) ele.RequireLong("size", path),
Width = ele.HasKey("width") ? ele.RequireInt("width", path) : null,
Height = ele.HasKey("height") ? ele.RequireInt("height", path) : null,
};
}).DistinctByKeyStable(static attachment => {
// Some Discord messages have duplicate attachments with the same id for unknown reasons.
return attachment.Id;

View File

@@ -1,30 +1,43 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<RootNamespace>DHT.Server</RootNamespace>
<Nullable>enable</Nullable>
<AssemblyName>DiscordHistoryTracker.Server</AssemblyName>
<PackageId>DiscordHistoryTrackerServer</PackageId>
<Authors>chylex</Authors>
<Company>DiscordHistoryTracker</Company>
<Product>DiscordHistoryTrackerServer</Product>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<DebugSymbols>true</DebugSymbols>
<DebugType>none</DebugType>
</PropertyGroup>
<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.Sqlite" Version="7.0.9" />
<PackageReference Include="Microsoft.Data.Sqlite" Version="8.0.0" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\Utils\Utils.csproj" />
</ItemGroup>
<ItemGroup>
<Compile Include="..\Version.cs" Link="Version.cs" />
</ItemGroup>
<ItemGroup>
<EmbeddedResource Include="../Resources/Tracker/bootstrap.js">
<LogicalName>Tracker\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
</EmbeddedResource>
<EmbeddedResource Include="../Resources/Tracker/scripts/**">
<LogicalName>Tracker\scripts\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/scripts/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
</EmbeddedResource>
<EmbeddedResource Include="../Resources/Tracker/styles/**">
<LogicalName>Tracker\styles\%(RecursiveDir)%(Filename)%(Extension)</LogicalName>
<Link>Resources/Tracker/styles/%(RecursiveDir)%(Filename)%(Extension)</Link>
<Visible>false</Visible>
</EmbeddedResource>
</ItemGroup>
</Project>

View File

@@ -61,14 +61,12 @@ public static class ServerLauncher {
}
}
private static void StartServerFromManagementThread(int port, string token, IDatabaseFile db) {
private static void StartServerFromManagementThread(ushort port, string token, IDatabaseFile db) {
Log.Info("Starting server on port " + port + "...");
void AddServices(IServiceCollection services) {
services.AddSingleton(typeof(IDatabaseFile), db);
services.AddSingleton(typeof(ServerParameters), new ServerParameters {
Token = token
});
services.AddSingleton(typeof(ServerParameters), new ServerParameters(port, token));
}
void SetKestrelOptions(KestrelServerOptions options) {
@@ -103,7 +101,7 @@ public static class ServerLauncher {
}
}
public static void Relaunch(int port, string token, IDatabaseFile db) {
public static void Relaunch(ushort port, string token, IDatabaseFile db) {
EnqueueMessage(new IMessage.StartServer(port, token, db));
}
@@ -113,11 +111,11 @@ public static class ServerLauncher {
private interface IMessage {
public sealed class StartServer : IMessage {
public int Port { get; }
public ushort Port { get; }
public string Token { get; }
public IDatabaseFile Db { get; }
public StartServer(int port, string token, IDatabaseFile db) {
public StartServer(ushort port, string token, IDatabaseFile db) {
this.Port = port;
this.Token = token;
this.Db = db;

View File

@@ -1,5 +1,3 @@
namespace DHT.Server.Service;
readonly struct ServerParameters {
public string Token { get; init; }
}
sealed record ServerParameters(ushort Port, string Token);

View File

@@ -24,7 +24,7 @@ sealed class Startup {
services.AddCors(static cors => {
cors.AddDefaultPolicy(static builder => {
builder.WithOrigins(AllowedOrigins).AllowCredentials().AllowAnyMethod().AllowAnyHeader();
builder.WithOrigins(AllowedOrigins).AllowCredentials().AllowAnyMethod().AllowAnyHeader().WithExposedHeaders("X-DHT");
});
});
}
@@ -34,17 +34,20 @@ sealed class Startup {
app.UseRouting();
app.UseCors();
app.UseEndpoints(endpoints => {
GetTrackingScriptEndpoint getTrackingScript = new (db, parameters);
endpoints.MapGet("/get-tracking-script", context => getTrackingScript.HandleGet(context));
TrackChannelEndpoint trackChannel = new (db, parameters);
endpoints.MapPost("/track-channel", async context => await trackChannel.HandlePost(context));
endpoints.MapPost("/track-channel", context => trackChannel.HandlePost(context));
TrackUsersEndpoint trackUsers = new (db, parameters);
endpoints.MapPost("/track-users", async context => await trackUsers.HandlePost(context));
endpoints.MapPost("/track-users", context => trackUsers.HandlePost(context));
TrackMessagesEndpoint trackMessages = new (db, parameters);
endpoints.MapPost("/track-messages", async context => await trackMessages.HandlePost(context));
endpoints.MapPost("/track-messages", context => trackMessages.HandlePost(context));
GetAttachmentEndpoint getAttachment = new (db, parameters);
endpoints.MapGet("/get-attachment/{url}", async context => await getAttachment.HandleGet(context));
endpoints.MapGet("/get-attachment/{url}", context => getAttachment.HandleGet(context));
});
}
}

View File

@@ -1,3 +1,4 @@
using System.Text;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
@@ -12,15 +13,15 @@ public static class HttpOutput {
}
}
public sealed class Json : IHttpOutput {
private readonly object? obj;
public sealed class Text : IHttpOutput {
private readonly string text;
public Json(object? obj) {
this.obj = obj;
public Text(string text) {
this.text = text;
}
public Task WriteTo(HttpResponse response) {
return response.WriteAsJsonAsync(obj);
return response.WriteAsync(text, Encoding.UTF8);
}
}

View File

@@ -0,0 +1,8 @@
using System.Text.Json;
using System.Text.Json.Serialization;
namespace DHT.Utils.Http;
[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase, GenerationMode = JsonSourceGenerationMode.Default)]
[JsonSerializable(typeof(JsonElement))]
public sealed partial class JsonElementContext : JsonSerializerContext {}

View File

@@ -1,27 +1,21 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<RootNamespace>DHT.Utils</RootNamespace>
<Nullable>enable</Nullable>
<AssemblyName>DiscordHistoryTracker.Utils</AssemblyName>
<PackageId>DiscordHistoryTrackerUtils</PackageId>
<Authors>chylex</Authors>
<Company>DiscordHistoryTracker</Company>
<Product>DiscordHistoryTrackerUtils</Product>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<DebugSymbols>true</DebugSymbols>
<DebugType>none</DebugType>
</PropertyGroup>
<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="JetBrains.Annotations" Version="2023.2.0" />
</ItemGroup>
<ItemGroup>
<Compile Include="..\Version.cs" Link="Version.cs" />
</ItemGroup>
</Project>

View File

@@ -8,5 +8,5 @@ using DHT.Utils;
namespace DHT.Utils;
static class Version {
public const string Tag = "37.2.0.0";
public const string Tag = "39.1.0.0";
}

View File

@@ -4,11 +4,11 @@ set list=win-x64 linux-x64 osx-x64
rmdir /S /Q bin
(for %%a in (%list%) do (
dotnet publish Desktop -c Release -r %%a -o ./bin/%%a -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true -p:PublishReadyToRun=false -p:PublishTrimmed=true -p:TrimMode=partial --self-contained true
dotnet publish Desktop -c Release -r %%a -o ./bin/%%a --self-contained true
powershell "Compress-Archive -Path ./bin/%%a/* -DestinationPath ./bin/%%a.zip -CompressionLevel Optimal"
))
dotnet publish Desktop -c Release -o ./bin/portable --self-contained false
dotnet publish Desktop -c Release -o ./bin/portable -p:PublishSingleFile=false -p:PublishTrimmed=false --self-contained false
powershell "Compress-Archive -Path ./bin/portable/* -DestinationPath ./bin/portable.zip -CompressionLevel Optimal"
echo Done

View File

@@ -17,9 +17,9 @@ rm -rf "./bin"
configurations=(win-x64 linux-x64 osx-x64)
for cfg in ${configurations[@]}; do
dotnet publish Desktop -c Release -r "$cfg" -o "./bin/$cfg" -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true -p:PublishReadyToRun=false -p:PublishTrimmed=true -p:TrimMode=partial --self-contained true
dotnet publish Desktop -c Release -r "$cfg" -o "./bin/$cfg" --self-contained true
makezip "$cfg"
done
dotnet publish Desktop -c Release -o "./bin/portable" --self-contained false
dotnet publish Desktop -c Release -o "./bin/portable" -p:PublishSingleFile=false -p:PublishTrimmed=false --self-contained false
makezip "portable"

Binary file not shown.

File diff suppressed because it is too large Load Diff

Binary file not shown.

22
lib/node_modules/commander/LICENSE generated vendored
View File

@@ -1,22 +0,0 @@
(The MIT License)
Copyright (c) 2011 TJ Holowaychuk <tj@vision-media.ca>
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

1110
lib/node_modules/commander/index.js generated vendored

File diff suppressed because it is too large Load Diff

View File

@@ -1,104 +0,0 @@
{
"_args": [
[
{
"raw": "commander@~2.9.0",
"scope": null,
"escapedName": "commander",
"name": "commander",
"rawSpec": "~2.9.0",
"spec": ">=2.9.0 <2.10.0",
"type": "range"
},
"C:\\Users\\Dan\\node_modules\\uglify-es"
]
],
"_from": "commander@>=2.9.0 <2.10.0",
"_id": "commander@2.9.0",
"_inCache": true,
"_location": "/commander",
"_nodeVersion": "0.12.7",
"_npmUser": {
"name": "zhiyelee",
"email": "zhiyelee@gmail.com"
},
"_npmVersion": "2.11.3",
"_phantomChildren": {},
"_requested": {
"raw": "commander@~2.9.0",
"scope": null,
"escapedName": "commander",
"name": "commander",
"rawSpec": "~2.9.0",
"spec": ">=2.9.0 <2.10.0",
"type": "range"
},
"_requiredBy": [
"/uglify-es"
],
"_resolved": "https://registry.npmjs.org/commander/-/commander-2.9.0.tgz",
"_shasum": "9c99094176e12240cb22d6c5146098400fe0f7d4",
"_shrinkwrap": null,
"_spec": "commander@~2.9.0",
"_where": "C:\\Users\\Dan\\node_modules\\uglify-es",
"author": {
"name": "TJ Holowaychuk",
"email": "tj@vision-media.ca"
},
"bugs": {
"url": "https://github.com/tj/commander.js/issues"
},
"dependencies": {
"graceful-readlink": ">= 1.0.0"
},
"description": "the complete solution for node.js command-line programs",
"devDependencies": {
"should": ">= 0.0.1",
"sinon": ">=1.17.1"
},
"directories": {},
"dist": {
"shasum": "9c99094176e12240cb22d6c5146098400fe0f7d4",
"tarball": "https://registry.npmjs.org/commander/-/commander-2.9.0.tgz"
},
"engines": {
"node": ">= 0.6.x"
},
"files": [
"index.js"
],
"gitHead": "b2aad7a8471d434593a85306aa73777a526e9f75",
"homepage": "https://github.com/tj/commander.js#readme",
"keywords": [
"command",
"option",
"parser"
],
"license": "MIT",
"main": "index",
"maintainers": [
{
"name": "tjholowaychuk",
"email": "tj@vision-media.ca"
},
{
"name": "somekittens",
"email": "rkoutnik@gmail.com"
},
{
"name": "zhiyelee",
"email": "zhiyelee@gmail.com"
}
],
"name": "commander",
"optionalDependencies": {},
"readme": "ERROR: No README data found!",
"repository": {
"type": "git",
"url": "git+https://github.com/tj/commander.js.git"
},
"scripts": {
"test": "make test"
},
"version": "2.9.0"
}

View File

@@ -1,22 +0,0 @@
The MIT License (MIT)
Copyright (c) 2015 Zhiye Li
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@@ -1,12 +0,0 @@
var fs = require('fs')
, lstat = fs.lstatSync;
exports.readlinkSync = function (p) {
if (lstat(p).isSymbolicLink()) {
return fs.readlinkSync(p);
} else {
return p;
}
};

View File

@@ -1,83 +0,0 @@
{
"_args": [
[
{
"raw": "graceful-readlink@>= 1.0.0",
"scope": null,
"escapedName": "graceful-readlink",
"name": "graceful-readlink",
"rawSpec": ">= 1.0.0",
"spec": ">=1.0.0",
"type": "range"
},
"C:\\Users\\Dan\\node_modules\\commander"
]
],
"_from": "graceful-readlink@>=1.0.0",
"_id": "graceful-readlink@1.0.1",
"_inCache": true,
"_location": "/graceful-readlink",
"_nodeVersion": "0.11.14",
"_npmUser": {
"name": "zhiyelee",
"email": "zhiyelee@gmail.com"
},
"_npmVersion": "2.1.17",
"_phantomChildren": {},
"_requested": {
"raw": "graceful-readlink@>= 1.0.0",
"scope": null,
"escapedName": "graceful-readlink",
"name": "graceful-readlink",
"rawSpec": ">= 1.0.0",
"spec": ">=1.0.0",
"type": "range"
},
"_requiredBy": [
"/commander"
],
"_resolved": "https://registry.npmjs.org/graceful-readlink/-/graceful-readlink-1.0.1.tgz",
"_shasum": "4cafad76bc62f02fa039b2f94e9a3dd3a391a725",
"_shrinkwrap": null,
"_spec": "graceful-readlink@>= 1.0.0",
"_where": "C:\\Users\\Dan\\node_modules\\commander",
"author": {
"name": "zhiyelee"
},
"bugs": {
"url": "https://github.com/zhiyelee/graceful-readlink/issues"
},
"dependencies": {},
"description": "graceful fs.readlink",
"devDependencies": {},
"directories": {},
"dist": {
"shasum": "4cafad76bc62f02fa039b2f94e9a3dd3a391a725",
"tarball": "https://registry.npmjs.org/graceful-readlink/-/graceful-readlink-1.0.1.tgz"
},
"gitHead": "f6655275bebef706fb63fd01b5f062a7052419a5",
"homepage": "https://github.com/zhiyelee/graceful-readlink",
"keywords": [
"fs.readlink",
"readlink"
],
"license": "MIT",
"main": "index.js",
"maintainers": [
{
"name": "zhiyelee",
"email": "zhiyelee@gmail.com"
}
],
"name": "graceful-readlink",
"optionalDependencies": {},
"readme": "ERROR: No README data found!",
"repository": {
"type": "git",
"url": "git://github.com/zhiyelee/graceful-readlink.git"
},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"version": "1.0.1"
}

28
lib/node_modules/source-map/LICENSE generated vendored
View File

@@ -1,28 +0,0 @@
Copyright (c) 2009-2011, Mozilla Foundation and contributors
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the names of the Mozilla Foundation nor the names of project
contributors may be used to endorse or promote products derived from this
software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

File diff suppressed because it is too large Load Diff

File diff suppressed because one or more lines are too long

View File

@@ -1,104 +0,0 @@
/* -*- Mode: js; js-indent-level: 2; -*- */
/*
* Copyright 2011 Mozilla Foundation and contributors
* Licensed under the New BSD license. See LICENSE or:
* http://opensource.org/licenses/BSD-3-Clause
*/
var util = require('./util');
var has = Object.prototype.hasOwnProperty;
/**
* A data structure which is a combination of an array and a set. Adding a new
* member is O(1), testing for membership is O(1), and finding the index of an
* element is O(1). Removing elements from the set is not supported. Only
* strings are supported for membership.
*/
function ArraySet() {
this._array = [];
this._set = Object.create(null);
}
/**
* Static method for creating ArraySet instances from an existing array.
*/
ArraySet.fromArray = function ArraySet_fromArray(aArray, aAllowDuplicates) {
var set = new ArraySet();
for (var i = 0, len = aArray.length; i < len; i++) {
set.add(aArray[i], aAllowDuplicates);
}
return set;
};
/**
* Return how many unique items are in this ArraySet. If duplicates have been
* added, than those do not count towards the size.
*
* @returns Number
*/
ArraySet.prototype.size = function ArraySet_size() {
return Object.getOwnPropertyNames(this._set).length;
};
/**
* Add the given string to this set.
*
* @param String aStr
*/
ArraySet.prototype.add = function ArraySet_add(aStr, aAllowDuplicates) {
var sStr = util.toSetString(aStr);
var isDuplicate = has.call(this._set, sStr);
var idx = this._array.length;
if (!isDuplicate || aAllowDuplicates) {
this._array.push(aStr);
}
if (!isDuplicate) {
this._set[sStr] = idx;
}
};
/**
* Is the given string a member of this set?
*
* @param String aStr
*/
ArraySet.prototype.has = function ArraySet_has(aStr) {
var sStr = util.toSetString(aStr);
return has.call(this._set, sStr);
};
/**
* What is the index of the given string in the array?
*
* @param String aStr
*/
ArraySet.prototype.indexOf = function ArraySet_indexOf(aStr) {
var sStr = util.toSetString(aStr);
if (has.call(this._set, sStr)) {
return this._set[sStr];
}
throw new Error('"' + aStr + '" is not in the set.');
};
/**
* What is the element at the given index?
*
* @param Number aIdx
*/
ArraySet.prototype.at = function ArraySet_at(aIdx) {
if (aIdx >= 0 && aIdx < this._array.length) {
return this._array[aIdx];
}
throw new Error('No element indexed by ' + aIdx);
};
/**
* Returns the array representation of this set (which has the proper indices
* indicated by indexOf). Note that this is a copy of the internal array used
* for storing the members so that no one can mess with internal state.
*/
ArraySet.prototype.toArray = function ArraySet_toArray() {
return this._array.slice();
};
exports.ArraySet = ArraySet;

View File

@@ -1,140 +0,0 @@
/* -*- Mode: js; js-indent-level: 2; -*- */
/*
* Copyright 2011 Mozilla Foundation and contributors
* Licensed under the New BSD license. See LICENSE or:
* http://opensource.org/licenses/BSD-3-Clause
*
* Based on the Base 64 VLQ implementation in Closure Compiler:
* https://code.google.com/p/closure-compiler/source/browse/trunk/src/com/google/debugging/sourcemap/Base64VLQ.java
*
* Copyright 2011 The Closure Compiler Authors. All rights reserved.
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials provided
* with the distribution.
* * Neither the name of Google Inc. nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
var base64 = require('./base64');
// A single base 64 digit can contain 6 bits of data. For the base 64 variable
// length quantities we use in the source map spec, the first bit is the sign,
// the next four bits are the actual value, and the 6th bit is the
// continuation bit. The continuation bit tells us whether there are more
// digits in this value following this digit.
//
// Continuation
// | Sign
// | |
// V V
// 101011
var VLQ_BASE_SHIFT = 5;
// binary: 100000
var VLQ_BASE = 1 << VLQ_BASE_SHIFT;
// binary: 011111
var VLQ_BASE_MASK = VLQ_BASE - 1;
// binary: 100000
var VLQ_CONTINUATION_BIT = VLQ_BASE;
/**
* Converts from a two-complement value to a value where the sign bit is
* placed in the least significant bit. For example, as decimals:
* 1 becomes 2 (10 binary), -1 becomes 3 (11 binary)
* 2 becomes 4 (100 binary), -2 becomes 5 (101 binary)
*/
function toVLQSigned(aValue) {
return aValue < 0
? ((-aValue) << 1) + 1
: (aValue << 1) + 0;
}
/**
* Converts to a two-complement value from a value where the sign bit is
* placed in the least significant bit. For example, as decimals:
* 2 (10 binary) becomes 1, 3 (11 binary) becomes -1
* 4 (100 binary) becomes 2, 5 (101 binary) becomes -2
*/
function fromVLQSigned(aValue) {
var isNegative = (aValue & 1) === 1;
var shifted = aValue >> 1;
return isNegative
? -shifted
: shifted;
}
/**
* Returns the base 64 VLQ encoded value.
*/
exports.encode = function base64VLQ_encode(aValue) {
var encoded = "";
var digit;
var vlq = toVLQSigned(aValue);
do {
digit = vlq & VLQ_BASE_MASK;
vlq >>>= VLQ_BASE_SHIFT;
if (vlq > 0) {
// There are still more digits in this value, so we must make sure the
// continuation bit is marked.
digit |= VLQ_CONTINUATION_BIT;
}
encoded += base64.encode(digit);
} while (vlq > 0);
return encoded;
};
/**
* Decodes the next base 64 VLQ value from the given string and returns the
* value and the rest of the string via the out parameter.
*/
exports.decode = function base64VLQ_decode(aStr, aIndex, aOutParam) {
var strLen = aStr.length;
var result = 0;
var shift = 0;
var continuation, digit;
do {
if (aIndex >= strLen) {
throw new Error("Expected more digits in base 64 VLQ value.");
}
digit = base64.decode(aStr.charCodeAt(aIndex++));
if (digit === -1) {
throw new Error("Invalid base64 digit: " + aStr.charAt(aIndex - 1));
}
continuation = !!(digit & VLQ_CONTINUATION_BIT);
digit &= VLQ_BASE_MASK;
result = result + (digit << shift);
shift += VLQ_BASE_SHIFT;
} while (continuation);
aOutParam.value = fromVLQSigned(result);
aOutParam.rest = aIndex;
};

View File

@@ -1,67 +0,0 @@
/* -*- Mode: js; js-indent-level: 2; -*- */
/*
* Copyright 2011 Mozilla Foundation and contributors
* Licensed under the New BSD license. See LICENSE or:
* http://opensource.org/licenses/BSD-3-Clause
*/
var intToCharMap = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'.split('');
/**
* Encode an integer in the range of 0 to 63 to a single base 64 digit.
*/
exports.encode = function (number) {
if (0 <= number && number < intToCharMap.length) {
return intToCharMap[number];
}
throw new TypeError("Must be between 0 and 63: " + number);
};
/**
* Decode a single base 64 character code digit to an integer. Returns -1 on
* failure.
*/
exports.decode = function (charCode) {
var bigA = 65; // 'A'
var bigZ = 90; // 'Z'
var littleA = 97; // 'a'
var littleZ = 122; // 'z'
var zero = 48; // '0'
var nine = 57; // '9'
var plus = 43; // '+'
var slash = 47; // '/'
var littleOffset = 26;
var numberOffset = 52;
// 0 - 25: ABCDEFGHIJKLMNOPQRSTUVWXYZ
if (bigA <= charCode && charCode <= bigZ) {
return (charCode - bigA);
}
// 26 - 51: abcdefghijklmnopqrstuvwxyz
if (littleA <= charCode && charCode <= littleZ) {
return (charCode - littleA + littleOffset);
}
// 52 - 61: 0123456789
if (zero <= charCode && charCode <= nine) {
return (charCode - zero + numberOffset);
}
// 62: +
if (charCode == plus) {
return 62;
}
// 63: /
if (charCode == slash) {
return 63;
}
// Invalid base64 digit.
return -1;
};

View File

@@ -1,111 +0,0 @@
/* -*- Mode: js; js-indent-level: 2; -*- */
/*
* Copyright 2011 Mozilla Foundation and contributors
* Licensed under the New BSD license. See LICENSE or:
* http://opensource.org/licenses/BSD-3-Clause
*/
exports.GREATEST_LOWER_BOUND = 1;
exports.LEAST_UPPER_BOUND = 2;
/**
* Recursive implementation of binary search.
*
* @param aLow Indices here and lower do not contain the needle.
* @param aHigh Indices here and higher do not contain the needle.
* @param aNeedle The element being searched for.
* @param aHaystack The non-empty array being searched.
* @param aCompare Function which takes two elements and returns -1, 0, or 1.
* @param aBias Either 'binarySearch.GREATEST_LOWER_BOUND' or
* 'binarySearch.LEAST_UPPER_BOUND'. Specifies whether to return the
* closest element that is smaller than or greater than the one we are
* searching for, respectively, if the exact element cannot be found.
*/
function recursiveSearch(aLow, aHigh, aNeedle, aHaystack, aCompare, aBias) {
// This function terminates when one of the following is true:
//
// 1. We find the exact element we are looking for.
//
// 2. We did not find the exact element, but we can return the index of
// the next-closest element.
//
// 3. We did not find the exact element, and there is no next-closest
// element than the one we are searching for, so we return -1.
var mid = Math.floor((aHigh - aLow) / 2) + aLow;
var cmp = aCompare(aNeedle, aHaystack[mid], true);
if (cmp === 0) {
// Found the element we are looking for.
return mid;
}
else if (cmp > 0) {
// Our needle is greater than aHaystack[mid].
if (aHigh - mid > 1) {
// The element is in the upper half.
return recursiveSearch(mid, aHigh, aNeedle, aHaystack, aCompare, aBias);
}
// The exact needle element was not found in this haystack. Determine if
// we are in termination case (3) or (2) and return the appropriate thing.
if (aBias == exports.LEAST_UPPER_BOUND) {
return aHigh < aHaystack.length ? aHigh : -1;
} else {
return mid;
}
}
else {
// Our needle is less than aHaystack[mid].
if (mid - aLow > 1) {
// The element is in the lower half.
return recursiveSearch(aLow, mid, aNeedle, aHaystack, aCompare, aBias);
}
// we are in termination case (3) or (2) and return the appropriate thing.
if (aBias == exports.LEAST_UPPER_BOUND) {
return mid;
} else {
return aLow < 0 ? -1 : aLow;
}
}
}
/**
* This is an implementation of binary search which will always try and return
* the index of the closest element if there is no exact hit. This is because
* mappings between original and generated line/col pairs are single points,
* and there is an implicit region between each of them, so a miss just means
* that you aren't on the very start of a region.
*
* @param aNeedle The element you are looking for.
* @param aHaystack The array that is being searched.
* @param aCompare A function which takes the needle and an element in the
* array and returns -1, 0, or 1 depending on whether the needle is less
* than, equal to, or greater than the element, respectively.
* @param aBias Either 'binarySearch.GREATEST_LOWER_BOUND' or
* 'binarySearch.LEAST_UPPER_BOUND'. Specifies whether to return the
* closest element that is smaller than or greater than the one we are
* searching for, respectively, if the exact element cannot be found.
* Defaults to 'binarySearch.GREATEST_LOWER_BOUND'.
*/
exports.search = function search(aNeedle, aHaystack, aCompare, aBias) {
if (aHaystack.length === 0) {
return -1;
}
var index = recursiveSearch(-1, aHaystack.length, aNeedle, aHaystack,
aCompare, aBias || exports.GREATEST_LOWER_BOUND);
if (index < 0) {
return -1;
}
// We have found either the exact element, or the next-closest element than
// the one we are searching for. However, there may be more than one such
// element. Make sure we always return the smallest of these.
while (index - 1 >= 0) {
if (aCompare(aHaystack[index], aHaystack[index - 1], true) !== 0) {
break;
}
--index;
}
return index;
};

View File

@@ -1,79 +0,0 @@
/* -*- Mode: js; js-indent-level: 2; -*- */
/*
* Copyright 2014 Mozilla Foundation and contributors
* Licensed under the New BSD license. See LICENSE or:
* http://opensource.org/licenses/BSD-3-Clause
*/
var util = require('./util');
/**
* Determine whether mappingB is after mappingA with respect to generated
* position.
*/
function generatedPositionAfter(mappingA, mappingB) {
// Optimized for most common case
var lineA = mappingA.generatedLine;
var lineB = mappingB.generatedLine;
var columnA = mappingA.generatedColumn;
var columnB = mappingB.generatedColumn;
return lineB > lineA || lineB == lineA && columnB >= columnA ||
util.compareByGeneratedPositionsInflated(mappingA, mappingB) <= 0;
}
/**
* A data structure to provide a sorted view of accumulated mappings in a
* performance conscious manner. It trades a neglibable overhead in general
* case for a large speedup in case of mappings being added in order.
*/
function MappingList() {
this._array = [];
this._sorted = true;
// Serves as infimum
this._last = {generatedLine: -1, generatedColumn: 0};
}
/**
* Iterate through internal items. This method takes the same arguments that
* `Array.prototype.forEach` takes.
*
* NOTE: The order of the mappings is NOT guaranteed.
*/
MappingList.prototype.unsortedForEach =
function MappingList_forEach(aCallback, aThisArg) {
this._array.forEach(aCallback, aThisArg);
};
/**
* Add the given source mapping.
*
* @param Object aMapping
*/
MappingList.prototype.add = function MappingList_add(aMapping) {
if (generatedPositionAfter(this._last, aMapping)) {
this._last = aMapping;
this._array.push(aMapping);
} else {
this._sorted = false;
this._array.push(aMapping);
}
};
/**
* Returns the flat, sorted array of mappings. The mappings are sorted by
* generated position.
*
* WARNING: This method returns internal data without copying, for
* performance. The return value must NOT be mutated, and should be treated as
* an immutable borrow. If you want to take ownership, you must make your own
* copy.
*/
MappingList.prototype.toArray = function MappingList_toArray() {
if (!this._sorted) {
this._array.sort(util.compareByGeneratedPositionsInflated);
this._sorted = true;
}
return this._array;
};
exports.MappingList = MappingList;

View File

@@ -1,114 +0,0 @@
/* -*- Mode: js; js-indent-level: 2; -*- */
/*
* Copyright 2011 Mozilla Foundation and contributors
* Licensed under the New BSD license. See LICENSE or:
* http://opensource.org/licenses/BSD-3-Clause
*/
// It turns out that some (most?) JavaScript engines don't self-host
// `Array.prototype.sort`. This makes sense because C++ will likely remain
// faster than JS when doing raw CPU-intensive sorting. However, when using a
// custom comparator function, calling back and forth between the VM's C++ and
// JIT'd JS is rather slow *and* loses JIT type information, resulting in
// worse generated code for the comparator function than would be optimal. In
// fact, when sorting with a comparator, these costs outweigh the benefits of
// sorting in C++. By using our own JS-implemented Quick Sort (below), we get
// a ~3500ms mean speed-up in `bench/bench.html`.
/**
* Swap the elements indexed by `x` and `y` in the array `ary`.
*
* @param {Array} ary
* The array.
* @param {Number} x
* The index of the first item.
* @param {Number} y
* The index of the second item.
*/
function swap(ary, x, y) {
var temp = ary[x];
ary[x] = ary[y];
ary[y] = temp;
}
/**
* Returns a random integer within the range `low .. high` inclusive.
*
* @param {Number} low
* The lower bound on the range.
* @param {Number} high
* The upper bound on the range.
*/
function randomIntInRange(low, high) {
return Math.round(low + (Math.random() * (high - low)));
}
/**
* The Quick Sort algorithm.
*
* @param {Array} ary
* An array to sort.
* @param {function} comparator
* Function to use to compare two items.
* @param {Number} p
* Start index of the array
* @param {Number} r
* End index of the array
*/
function doQuickSort(ary, comparator, p, r) {
// If our lower bound is less than our upper bound, we (1) partition the
// array into two pieces and (2) recurse on each half. If it is not, this is
// the empty array and our base case.
if (p < r) {
// (1) Partitioning.
//
// The partitioning chooses a pivot between `p` and `r` and moves all
// elements that are less than or equal to the pivot to the before it, and
// all the elements that are greater than it after it. The effect is that
// once partition is done, the pivot is in the exact place it will be when
// the array is put in sorted order, and it will not need to be moved
// again. This runs in O(n) time.
// Always choose a random pivot so that an input array which is reverse
// sorted does not cause O(n^2) running time.
var pivotIndex = randomIntInRange(p, r);
var i = p - 1;
swap(ary, pivotIndex, r);
var pivot = ary[r];
// Immediately after `j` is incremented in this loop, the following hold
// true:
//
// * Every element in `ary[p .. i]` is less than or equal to the pivot.
//
// * Every element in `ary[i+1 .. j-1]` is greater than the pivot.
for (var j = p; j < r; j++) {
if (comparator(ary[j], pivot) <= 0) {
i += 1;
swap(ary, i, j);
}
}
swap(ary, i + 1, j);
var q = i + 1;
// (2) Recurse on each half.
doQuickSort(ary, comparator, p, q - 1);
doQuickSort(ary, comparator, q + 1, r);
}
}
/**
* Sort the given array in-place with the given comparator function.
*
* @param {Array} ary
* An array to sort.
* @param {function} comparator
* Function to use to compare two items.
*/
exports.quickSort = function (ary, comparator) {
doQuickSort(ary, comparator, 0, ary.length - 1);
};

File diff suppressed because it is too large Load Diff

View File

@@ -1,404 +0,0 @@
/* -*- Mode: js; js-indent-level: 2; -*- */
/*
* Copyright 2011 Mozilla Foundation and contributors
* Licensed under the New BSD license. See LICENSE or:
* http://opensource.org/licenses/BSD-3-Clause
*/
var base64VLQ = require('./base64-vlq');
var util = require('./util');
var ArraySet = require('./array-set').ArraySet;
var MappingList = require('./mapping-list').MappingList;
/**
* An instance of the SourceMapGenerator represents a source map which is
* being built incrementally. You may pass an object with the following
* properties:
*
* - file: The filename of the generated source.
* - sourceRoot: A root for all relative URLs in this source map.
*/
function SourceMapGenerator(aArgs) {
if (!aArgs) {
aArgs = {};
}
this._file = util.getArg(aArgs, 'file', null);
this._sourceRoot = util.getArg(aArgs, 'sourceRoot', null);
this._skipValidation = util.getArg(aArgs, 'skipValidation', false);
this._sources = new ArraySet();
this._names = new ArraySet();
this._mappings = new MappingList();
this._sourcesContents = null;
}
SourceMapGenerator.prototype._version = 3;
/**
* Creates a new SourceMapGenerator based on a SourceMapConsumer
*
* @param aSourceMapConsumer The SourceMap.
*/
SourceMapGenerator.fromSourceMap =
function SourceMapGenerator_fromSourceMap(aSourceMapConsumer) {
var sourceRoot = aSourceMapConsumer.sourceRoot;
var generator = new SourceMapGenerator({
file: aSourceMapConsumer.file,
sourceRoot: sourceRoot
});
aSourceMapConsumer.eachMapping(function (mapping) {
var newMapping = {
generated: {
line: mapping.generatedLine,
column: mapping.generatedColumn
}
};
if (mapping.source != null) {
newMapping.source = mapping.source;
if (sourceRoot != null) {
newMapping.source = util.relative(sourceRoot, newMapping.source);
}
newMapping.original = {
line: mapping.originalLine,
column: mapping.originalColumn
};
if (mapping.name != null) {
newMapping.name = mapping.name;
}
}
generator.addMapping(newMapping);
});
aSourceMapConsumer.sources.forEach(function (sourceFile) {
var content = aSourceMapConsumer.sourceContentFor(sourceFile);
if (content != null) {
generator.setSourceContent(sourceFile, content);
}
});
return generator;
};
/**
* Add a single mapping from original source line and column to the generated
* source's line and column for this source map being created. The mapping
* object should have the following properties:
*
* - generated: An object with the generated line and column positions.
* - original: An object with the original line and column positions.
* - source: The original source file (relative to the sourceRoot).
* - name: An optional original token name for this mapping.
*/
SourceMapGenerator.prototype.addMapping =
function SourceMapGenerator_addMapping(aArgs) {
var generated = util.getArg(aArgs, 'generated');
var original = util.getArg(aArgs, 'original', null);
var source = util.getArg(aArgs, 'source', null);
var name = util.getArg(aArgs, 'name', null);
if (!this._skipValidation) {
this._validateMapping(generated, original, source, name);
}
if (source != null) {
source = String(source);
if (!this._sources.has(source)) {
this._sources.add(source);
}
}
if (name != null) {
name = String(name);
if (!this._names.has(name)) {
this._names.add(name);
}
}
this._mappings.add({
generatedLine: generated.line,
generatedColumn: generated.column,
originalLine: original != null && original.line,
originalColumn: original != null && original.column,
source: source,
name: name
});
};
/**
* Set the source content for a source file.
*/
SourceMapGenerator.prototype.setSourceContent =
function SourceMapGenerator_setSourceContent(aSourceFile, aSourceContent) {
var source = aSourceFile;
if (this._sourceRoot != null) {
source = util.relative(this._sourceRoot, source);
}
if (aSourceContent != null) {
// Add the source content to the _sourcesContents map.
// Create a new _sourcesContents map if the property is null.
if (!this._sourcesContents) {
this._sourcesContents = Object.create(null);
}
this._sourcesContents[util.toSetString(source)] = aSourceContent;
} else if (this._sourcesContents) {
// Remove the source file from the _sourcesContents map.
// If the _sourcesContents map is empty, set the property to null.
delete this._sourcesContents[util.toSetString(source)];
if (Object.keys(this._sourcesContents).length === 0) {
this._sourcesContents = null;
}
}
};
/**
* Applies the mappings of a sub-source-map for a specific source file to the
* source map being generated. Each mapping to the supplied source file is
* rewritten using the supplied source map. Note: The resolution for the
* resulting mappings is the minimium of this map and the supplied map.
*
* @param aSourceMapConsumer The source map to be applied.
* @param aSourceFile Optional. The filename of the source file.
* If omitted, SourceMapConsumer's file property will be used.
* @param aSourceMapPath Optional. The dirname of the path to the source map
* to be applied. If relative, it is relative to the SourceMapConsumer.
* This parameter is needed when the two source maps aren't in the same
* directory, and the source map to be applied contains relative source
* paths. If so, those relative source paths need to be rewritten
* relative to the SourceMapGenerator.
*/
SourceMapGenerator.prototype.applySourceMap =
function SourceMapGenerator_applySourceMap(aSourceMapConsumer, aSourceFile, aSourceMapPath) {
var sourceFile = aSourceFile;
// If aSourceFile is omitted, we will use the file property of the SourceMap
if (aSourceFile == null) {
if (aSourceMapConsumer.file == null) {
throw new Error(
'SourceMapGenerator.prototype.applySourceMap requires either an explicit source file, ' +
'or the source map\'s "file" property. Both were omitted.'
);
}
sourceFile = aSourceMapConsumer.file;
}
var sourceRoot = this._sourceRoot;
// Make "sourceFile" relative if an absolute Url is passed.
if (sourceRoot != null) {
sourceFile = util.relative(sourceRoot, sourceFile);
}
// Applying the SourceMap can add and remove items from the sources and
// the names array.
var newSources = new ArraySet();
var newNames = new ArraySet();
// Find mappings for the "sourceFile"
this._mappings.unsortedForEach(function (mapping) {
if (mapping.source === sourceFile && mapping.originalLine != null) {
// Check if it can be mapped by the source map, then update the mapping.
var original = aSourceMapConsumer.originalPositionFor({
line: mapping.originalLine,
column: mapping.originalColumn
});
if (original.source != null) {
// Copy mapping
mapping.source = original.source;
if (aSourceMapPath != null) {
mapping.source = util.join(aSourceMapPath, mapping.source)
}
if (sourceRoot != null) {
mapping.source = util.relative(sourceRoot, mapping.source);
}
mapping.originalLine = original.line;
mapping.originalColumn = original.column;
if (original.name != null) {
mapping.name = original.name;
}
}
}
var source = mapping.source;
if (source != null && !newSources.has(source)) {
newSources.add(source);
}
var name = mapping.name;
if (name != null && !newNames.has(name)) {
newNames.add(name);
}
}, this);
this._sources = newSources;
this._names = newNames;
// Copy sourcesContents of applied map.
aSourceMapConsumer.sources.forEach(function (sourceFile) {
var content = aSourceMapConsumer.sourceContentFor(sourceFile);
if (content != null) {
if (aSourceMapPath != null) {
sourceFile = util.join(aSourceMapPath, sourceFile);
}
if (sourceRoot != null) {
sourceFile = util.relative(sourceRoot, sourceFile);
}
this.setSourceContent(sourceFile, content);
}
}, this);
};
/**
* A mapping can have one of the three levels of data:
*
* 1. Just the generated position.
* 2. The Generated position, original position, and original source.
* 3. Generated and original position, original source, as well as a name
* token.
*
* To maintain consistency, we validate that any new mapping being added falls
* in to one of these categories.
*/
SourceMapGenerator.prototype._validateMapping =
function SourceMapGenerator_validateMapping(aGenerated, aOriginal, aSource,
aName) {
if (aGenerated && 'line' in aGenerated && 'column' in aGenerated
&& aGenerated.line > 0 && aGenerated.column >= 0
&& !aOriginal && !aSource && !aName) {
// Case 1.
return;
}
else if (aGenerated && 'line' in aGenerated && 'column' in aGenerated
&& aOriginal && 'line' in aOriginal && 'column' in aOriginal
&& aGenerated.line > 0 && aGenerated.column >= 0
&& aOriginal.line > 0 && aOriginal.column >= 0
&& aSource) {
// Cases 2 and 3.
return;
}
else {
throw new Error('Invalid mapping: ' + JSON.stringify({
generated: aGenerated,
source: aSource,
original: aOriginal,
name: aName
}));
}
};
/**
* Serialize the accumulated mappings in to the stream of base 64 VLQs
* specified by the source map format.
*/
SourceMapGenerator.prototype._serializeMappings =
function SourceMapGenerator_serializeMappings() {
var previousGeneratedColumn = 0;
var previousGeneratedLine = 1;
var previousOriginalColumn = 0;
var previousOriginalLine = 0;
var previousName = 0;
var previousSource = 0;
var result = '';
var next;
var mapping;
var nameIdx;
var sourceIdx;
var mappings = this._mappings.toArray();
for (var i = 0, len = mappings.length; i < len; i++) {
mapping = mappings[i];
next = ''
if (mapping.generatedLine !== previousGeneratedLine) {
previousGeneratedColumn = 0;
while (mapping.generatedLine !== previousGeneratedLine) {
next += ';';
previousGeneratedLine++;
}
}
else {
if (i > 0) {
if (!util.compareByGeneratedPositionsInflated(mapping, mappings[i - 1])) {
continue;
}
next += ',';
}
}
next += base64VLQ.encode(mapping.generatedColumn
- previousGeneratedColumn);
previousGeneratedColumn = mapping.generatedColumn;
if (mapping.source != null) {
sourceIdx = this._sources.indexOf(mapping.source);
next += base64VLQ.encode(sourceIdx - previousSource);
previousSource = sourceIdx;
// lines are stored 0-based in SourceMap spec version 3
next += base64VLQ.encode(mapping.originalLine - 1
- previousOriginalLine);
previousOriginalLine = mapping.originalLine - 1;
next += base64VLQ.encode(mapping.originalColumn
- previousOriginalColumn);
previousOriginalColumn = mapping.originalColumn;
if (mapping.name != null) {
nameIdx = this._names.indexOf(mapping.name);
next += base64VLQ.encode(nameIdx - previousName);
previousName = nameIdx;
}
}
result += next;
}
return result;
};
SourceMapGenerator.prototype._generateSourcesContent =
function SourceMapGenerator_generateSourcesContent(aSources, aSourceRoot) {
return aSources.map(function (source) {
if (!this._sourcesContents) {
return null;
}
if (aSourceRoot != null) {
source = util.relative(aSourceRoot, source);
}
var key = util.toSetString(source);
return Object.prototype.hasOwnProperty.call(this._sourcesContents, key)
? this._sourcesContents[key]
: null;
}, this);
};
/**
* Externalize the source map.
*/
SourceMapGenerator.prototype.toJSON =
function SourceMapGenerator_toJSON() {
var map = {
version: this._version,
sources: this._sources.toArray(),
names: this._names.toArray(),
mappings: this._serializeMappings()
};
if (this._file != null) {
map.file = this._file;
}
if (this._sourceRoot != null) {
map.sourceRoot = this._sourceRoot;
}
if (this._sourcesContents) {
map.sourcesContent = this._generateSourcesContent(map.sources, map.sourceRoot);
}
return map;
};
/**
* Render the source map being generated to a string.
*/
SourceMapGenerator.prototype.toString =
function SourceMapGenerator_toString() {
return JSON.stringify(this.toJSON());
};
exports.SourceMapGenerator = SourceMapGenerator;

Some files were not shown because too many files have changed in this diff Show More