The normalizing algorithm has been rewritten to be more efficient and hopefully more reliable. The new algorithm de-constructs each line after converting it to the system's native line ending. Then it searches for the key and returns value. The rewrite also normalizes line endings to match the system's within the entry itself before returning the final output. This should make things more stable and predictable.

- Skipping comments is a little buggy.
- Multiline parsing with the v2 format is still unpredictable.
This commit is contained in:
Tony Bark 2020-12-11 01:54:20 -05:00 committed by GitHub
commit e6c8bc914a
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
18 changed files with 344 additions and 182 deletions

17
.github/ISSUE_TEMPLATE.md vendored Normal file
View file

@ -0,0 +1,17 @@
## Expected Behavior
## Actual Behavior
## Steps to Reproduce the Problem
1.
1.
1.
## Specifications
- Version:
- Platform:
- Operating System:

7
.github/PULL_REQUEST_TEMPLATE.md vendored Normal file
View file

@ -0,0 +1,7 @@
# Fixes
## Proposed Changes
-
-
-

View file

@ -2,9 +2,9 @@ name: .NET
on:
push:
branches: [master]
branches: [master, develop]
pull_request:
branches: [master]
branches: [master, develop]
jobs:
build:

View file

@ -18,8 +18,8 @@ jobs:
run: dotnet restore
- name: Package
run: dotnet pack --no-restore -o .
# - name: Publish to Nuget
# env:
# NUGET_KEY: ${{ secrets.NUGET_KEY }}
# NUGET_API: https://api.nuget.org/v3/index.json
# run: dotnet nuget push CSTNet.*.nupkg -k $NUGET_KEY -s $NUGET_API
- name: Publish to Nuget
env:
NUGET_KEY: ${{ secrets.NUGET_KEY }}
NUGET_API: https://api.nuget.org/v3/index.json
run: dotnet nuget push CSTNet.*.nupkg -k $NUGET_KEY -s $NUGET_API

33
.gitignore vendored
View file

@ -1,5 +1,7 @@
# Created by https://www.gitignore.io/api/rider,linux,macos,backup,windows,visualstudio,visualstudiocode
# Edit at https://www.gitignore.io/?templates=rider,linux,macos,backup,windows,visualstudio,visualstudiocode
# File created using '.gitignore Generator' for Visual Studio Code: https://bit.ly/vscode-gig
# Created by https://www.toptal.com/developers/gitignore/api/windows,visualstudiocode,visualstudio,rider,macos,linux,backup
# Edit at https://www.toptal.com/developers/gitignore?templates=windows,visualstudiocode,visualstudio,rider,macos,linux,backup
### Backup ###
*.bak
@ -32,6 +34,7 @@
# Icon must end with two \r
Icon
# Thumbnails
._*
@ -52,17 +55,19 @@ Temporary Items
.apdisk
### Rider ###
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
# User-specific stuff
.idea/**
**/.idea/**
# Gradle and Maven with auto-import
# When using Gradle or Maven with auto-import, you should exclude module files,
# since they will be recreated, and may cause churn. Uncomment if using
# auto-import.
# .idea/artifacts
# .idea/compiler.xml
# .idea/jarRepositories.xml
# .idea/modules.xml
# .idea/*.iml
# .idea/modules
@ -104,14 +109,14 @@ fabric.properties
### VisualStudioCode ###
.vscode/*
!.vscode/settings.json
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
*.code-workspace
### VisualStudioCode Patch ###
# Ignore all local history of files
.history
.ionide
### Windows ###
# Windows thumbnail cache files
@ -171,6 +176,7 @@ bld/
[Bb]in/
[Oo]bj/
[Ll]og/
[Ll]ogs/
# Visual Studio 2015/2017 cache/options directory
.vs/
@ -210,6 +216,7 @@ StyleCopReport.xml
*_p.c
*_h.h
*.ilk
*.meta
*.obj
*.iobj
*.pch
@ -266,9 +273,6 @@ _ReSharper*/
*.[Rr]e[Ss]harper
*.DotSettings.user
# JustCode is a .NET coding add-in
.JustCode
# TeamCity is a build add-in
_TeamCity*
@ -279,6 +283,9 @@ _TeamCity*
.axoCover/*
!.axoCover/settings.json
# Coverlet is a free, cross platform Code Coverage Tool
coverage*[.json, .xml, .info]
# Visual Studio code coverage results
*.coverage
*.coveragexml
@ -487,4 +494,10 @@ healthchecksdb
# Backup folder for Package Reference Convert tool in Visual Studio 2017
MigrationBackup/
# End of https://www.gitignore.io/api/rider,linux,macos,backup,windows,visualstudio,visualstudiocode
# Ionide (cross platform F# VS Code tools) working folder
.ionide/
# End of https://www.toptal.com/developers/gitignore/api/windows,visualstudiocode,visualstudio,rider,macos,linux,backup
# Custom rules (everything added below won't be overriden by 'Generate .gitignore File' if you use 'Update' option)

View file

@ -1,40 +0,0 @@
// This project is licensed under the MIT license.
using System;
using System.IO;
using Xunit;
namespace CSTNet.Tests
{
public class BasicTests
{
string CSTFile(string cst, string key)
{
var path = Path.Combine(AppContext.BaseDirectory, cst);
var file = File.ReadAllText(path);
return CaretSeparatedText.Parse(file, key);
}
[Theory]
[InlineData(1, "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin ac dictum orci, at tincidunt nulla. Donec aliquet, eros non interdum posuere, ipsum sapien molestie nunc, nec facilisis libero ipsum et risus. In sed lorem vel ipsum placerat viverra.")]
[InlineData(4, @"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce justo dui, rhoncus a pulvinar sit amet, fermentum vitae lorem. Maecenas nec nisi sit amet eros rutrum congue. In sagittis suscipit arcu, ac vestibulum nunc feugiat volutpat.
Vivamus consequat velit dui, sit amet rhoncus dui malesuada a. Maecenas hendrerit commodo mi et scelerisque. Cras pharetra ultrices aliquam. Praesent ac efficitur magna, vitae scelerisque metus.")]
public void V1Test(int key, string expected)
{
var actual = CSTFile("v1.cst", key.ToString());
Assert.Equal(expected, actual);
}
[Theory]
[InlineData("Singleline", "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed ultricies nulla eu tortor mattis, dictum posuere lacus ornare. Maecenas a massa in ligula finibus luctus eu vitae nibh. Proin imperdiet dapibus mauris quis placerat.")]
[InlineData("Multiline", @"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc gravida nunc non justo pretium consectetur. Sed tempus libero ac ligula aliquam elementum. Duis vitae interdum leo. Sed semper nulla %1 a lectus dictum dictum.
Quisque vehicula, nisi ut scelerisque sodales, nisi ipsum sodales ipsum, in rutrum tellus lacus sed nibh. Etiam mauris velit, elementum sed placerat et, elementum et tellus. Duis vitae elit fermentum, viverra lorem in, lobortis elit.")]
public void V2Test(string key, string expected)
{
var actual = CSTFile("v2.cst", key);
Assert.Equal(expected, actual);
}
}
}

16
CSTNet.Tests/CSTHelper.cs Normal file
View file

@ -0,0 +1,16 @@
using System;
using System.IO;
namespace CSTNet.Tests
{
static class CSTHelper
{
public static string CSTFile(string cst, string key)
{
var path = Path.Combine(AppContext.BaseDirectory, cst);
var file = File.ReadAllText(path);
return CaretSeparatedText.Parse(file, key);
}
}
}

View file

@ -0,0 +1,25 @@
using System;
using Xunit;
namespace CSTNet.Tests
{
public class MultilineTests
{
[Fact]
public void MiltilineV1()
{
var four = 4;
var expected = $"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce justo dui, rhoncus a pulvinar sit amet, fermentum vitae lorem. Maecenas nec nisi sit amet eros rutrum congue. In sagittis suscipit arcu, ac vestibulum nunc feugiat volutpat.{Environment.NewLine}{Environment.NewLine}Vivamus consequat velit dui, sit amet rhoncus dui malesuada a. Maecenas hendrerit commodo mi et scelerisque. Cras pharetra ultrices aliquam. Praesent ac efficitur magna, vitae scelerisque metus.";
var actual = CSTHelper.CSTFile("v1.cst", four.ToString());
Assert.Equal(expected, actual);
}
[Fact]
public void MiltilineV2()
{
var expected = $"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc gravida nunc non justo pretium consectetur. Sed tempus libero ac ligula aliquam elementum. Duis vitae interdum leo. Sed semper nulla %1 a lectus dictum dictum.{Environment.NewLine}{Environment.NewLine}Quisque vehicula, nisi ut scelerisque sodales, nisi ipsum sodales ipsum, in rutrum tellus lacus sed nibh. Etiam mauris velit, elementum sed placerat et, elementum et tellus. Duis vitae elit fermentum, viverra lorem in, lobortis elit.";
var actual = CSTHelper.CSTFile("v2.cst", "Multiline");
Assert.Equal(expected, actual);
}
}
}

View file

@ -1,35 +1,34 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<IsPackable>false</IsPackable>
</PropertyGroup>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.7.1" />
<PackageReference Include="xunit" Version="2.4.1" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="coverlet.collector" Version="1.3.0">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.7.1" />
<PackageReference Include="xunit" Version="2.4.1" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="coverlet.collector" Version="1.3.0">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\CSTNet\QuickFennec.CST.csproj" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\CSTNet\QuickFennec.CST.csproj" />
</ItemGroup>
<ItemGroup>
<None Update="v1.cst">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="v2.cst">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
</ItemGroup>
<ItemGroup>
<None Update="v1.cst">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="v2.cst">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>

View file

@ -0,0 +1,25 @@
// This project is licensed under the MIT license.
using Xunit;
namespace CSTNet.Tests
{
public class SingleLineTests
{
[Theory]
[InlineData(1, @"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin ac dictum orci, at tincidunt nulla. Donec aliquet, eros non interdum posuere, ipsum sapien molestie nunc, nec facilisis libero ipsum et risus. In sed lorem vel ipsum placerat viverra.")]
[InlineData(3, @"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam venenatis ac odio ut pretium. Interdum et malesuada fames ac ante ipsum primis in faucibus. Donec semper turpis tempor, bibendum sapien at, blandit neque. Vivamus hendrerit imperdiet elit, vel sollicitudin nulla luctus vel. Vivamus nisl quam, feugiat a diam aliquam, iaculis vestibulum nunc. Maecenas euismod leo enim, faucibus ultrices ipsum semper eu. Praesent ullamcorper justo at maximus ultricies.")]
public void V1Test(int key, string expected)
{
var actual = CSTHelper.CSTFile("v1.cst", key.ToString());
Assert.Equal(expected, actual);
}
[Theory]
[InlineData("Singleline", "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed ultricies nulla eu tortor mattis, dictum posuere lacus ornare. Maecenas a massa in ligula finibus luctus eu vitae nibh. Proin imperdiet dapibus mauris quis placerat.")]
public void V2Test(string key, string expected)
{
var actual = CSTHelper.CSTFile("v2.cst", key);
Assert.Equal(expected, actual);
}
}
}

View file

@ -3,6 +3,6 @@
Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Nunc vel dictum eros, vitae mattis risus. Curabitur eget nisi interdum, euismod nisl in, fermentum turpis. Morbi a feugiat lacus. Duis ligula felis, commodo quis sodales ac, congue sit amet tortor. Sed vulputate, velit id interdum convallis, purus nisl interdum lorem, sit amet aliquam lacus sapien ac neque. Proin sit amet ultricies mi.^
3 ^Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam venenatis ac odio ut pretium. Interdum et malesuada fames ac ante ipsum primis in faucibus. Donec semper turpis tempor, bibendum sapien at, blandit neque. Vivamus hendrerit imperdiet elit, vel sollicitudin nulla luctus vel. Vivamus nisl quam, feugiat a diam aliquam, iaculis vestibulum nunc. Maecenas euismod leo enim, faucibus ultrices ipsum semper eu. Praesent ullamcorper justo at maximus ultricies.^
4^Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce justo dui, rhoncus a pulvinar sit amet, fermentum vitae lorem. Maecenas nec nisi sit amet eros rutrum congue. In sagittis suscipit arcu, ac vestibulum nunc feugiat volutpat.
4 ^Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce justo dui, rhoncus a pulvinar sit amet, fermentum vitae lorem. Maecenas nec nisi sit amet eros rutrum congue. In sagittis suscipit arcu, ac vestibulum nunc feugiat volutpat.
Vivamus consequat velit dui, sit amet rhoncus dui malesuada a. Maecenas hendrerit commodo mi et scelerisque. Cras pharetra ultrices aliquam. Praesent ac efficitur magna, vitae scelerisque metus.^

View file

@ -8,6 +8,7 @@ EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{CCFCE2DB-C18F-4D88-B025-19ED62BD2A1D}"
ProjectSection(SolutionItems) = preProject
.editorconfig = .editorconfig
changelog.md = changelog.md
README.md = README.md
EndProjectSection
EndProject

View file

@ -6,6 +6,12 @@ namespace CSTNet
{
public static class CaretSeparatedText
{
const char CARET = '^';
static readonly string _lf = "\u000A";
static readonly string _cr = "\u000D";
static readonly string _crlf = "\u000D\u000A";
static readonly string _ls = "\u2028";
/// <summary>
/// Gets the value from the integer-based key.
/// </summary>
@ -13,7 +19,7 @@ namespace CSTNet
public static string Parse(string content, int key)
{
var entries = NormalizeEntries(content);
return GetEntry(entries, $"{key}");
return GetEntry(entries, key.ToString());
}
/// <summary>
@ -32,52 +38,79 @@ namespace CSTNet
/// <remarks>This stage ensures there are no crashes during parsing.</remarks>
static IEnumerable<string> NormalizeEntries(string content)
{
var lineBreaks = new string[]
/*
I tried putting the end carets with the different
line endings in with the split function but it didn't work
*/
if (!content.Contains($"{CARET}{Environment.NewLine}"))
{
"^\u000A", // LF
"^\u000D", // CR
"^\u000D\u000A", // CR+LF
"^\u2028" // LS
};
if (content.Contains($"{CARET}{_lf}"))
content = content.Replace($"{CARET}{_lf}",
$"{CARET}{Environment.NewLine}");
foreach (var line in lineBreaks)
{
var eol = Environment.NewLine; // System's line break
if (content.Contains($"{CARET}{_cr}"))
content = content.Replace($"{CARET}{_cr}",
$"{CARET}{Environment.NewLine}");
// If the new line matches the system's, do nothing
if (line.Contains(eol))
continue;
if (content.Contains($"{CARET}{_crlf}"))
content = content.Replace($"{CARET}{_crlf}",
$"{CARET}{Environment.NewLine}");
content.Replace(line, eol);
if (content.Contains($"{CARET}{_ls}"))
content = content.Replace($"{CARET}{_ls}",
$"{CARET}{Environment.NewLine}");
}
return content.Split(lineBreaks, StringSplitOptions.RemoveEmptyEntries);
var entries = content.Split(new[] { $"{CARET}{Environment.NewLine}" },
StringSplitOptions.RemoveEmptyEntries);
var newContent = new List<string>();
foreach (var entry in entries)
{
// Skip comments
if (entry.StartsWith(@"//") || entry.StartsWith("#") ||
entry.StartsWith("/*") || entry.EndsWith("*/"))
continue;
newContent.Add(entry);
}
return newContent;
}
// TODO: support argument parameters
static string GetEntry(IEnumerable<string> entries, string key)
{
var translation = "[ENTRY NOT FOUND]";
// Search through array
// Search through list
foreach (var entry in entries)
{
// Locate index, trim carets and return translation
if (!entry.StartsWith(key))
continue;
const char caret = '^';
var startIndex = entry.IndexOf(caret.ToString(),
StringComparison.OrdinalIgnoreCase);
var startIndex = entry.IndexOf(CARET);
var line = entry.Substring(startIndex);
translation = line.Trim(caret);
if (!line.Contains(Environment.NewLine))
{
if (line.Contains(_lf))
line = line.Replace(_lf, Environment.NewLine);
if (line.Contains(_cr))
line = line.Replace(_cr, Environment.NewLine);
if (line.Contains(_crlf))
line = line.Replace(_crlf, Environment.NewLine);
if (line.Contains(_ls))
line = line.Replace(_ls, Environment.NewLine);
}
return line.TrimStart(CARET).TrimEnd(CARET);
}
return translation;
return "[ENTRY NOT FOUND]";
}
}
}

View file

@ -2,12 +2,13 @@
<PropertyGroup>
<TargetFramework>netstandard2.0</TargetFramework>
<Version>1.0</Version>
<Version>1.0.1</Version>
<Authors>Tony Bark</Authors>
<PackageDescription>Caret-Separated Text (or CST) is a key-value pair format represented by numbers or words as keys and the value is the string enclosed between carets (^) that contains the contents. CST.NET is a library for prasing the CST format.</PackageDescription>
<RepositoryUrl>https://github.com/tonytins/cstnet</RepositoryUrl>
<PackageLicenseExpression>MIT</PackageLicenseExpression>
<Product>CST.Net</Product>
<AssemblyName>CSTNet</AssemblyName>
</PropertyGroup>
</Project>

View file

@ -1,12 +1,15 @@
# CST.NET
# QuickFennec.CST
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg)](code_of_conduct.md)
Caret-Separated Text (or CST) is a key-value pair format represented by numbers or words as keys and the value is the string enclosed between carets (^) that contains the contents. CST.NET is a library for parsing the CST format.
Caret-Separated Text (or CST) is a key-value pair format represented by numbers or words as keys and the value is the string enclosed between carets (^) that contains the contents. Any text which is not enclosed with carets is considered a comment and ignored. Neither strings nor comments may use the caret character.
QuickFennec.CST is a library for parsing the CST format.
## Usage
```csharp
#r "nuget:CSTNet,1.0.1"
using System;
using System.IO;
using CSTNet;
@ -19,15 +22,20 @@ Console.WriteLine(example);
In production, CST files were used in The Sims Online to provide translations. Each translation was split into their respective directories:
- ``uitext/english.dir/hints/toolbar.cst``
- ``uitext/swedish.dir/hints/toolbar.cst``
- ``uitext/english.dir/misc/_154_miscstrings.cst``
- ``uitext/swedish.dir/misc/_154_miscstrings.cst``
As long as the key remained the same, it didn't matter what directory the file was located in.
QuickFennec.CST only provides the basic parsing functionality.
## To-do
- [ ] Support for arguments (e.g. ``%1``)
## Known issues
- Skipping comments is a little buggy.
- Multiline parsing with the v2 format is still unpredictable.
## Requirements
### Prerequisites

22
changelog.md Normal file
View file

@ -0,0 +1,22 @@
# Change Log
## 1.0.1
Despite only being a point release, this includes a major refinement to the normalizing algorithm.
### Rewrote normalizing algorithm
The normalizing algorithm has been rewritten to be more efficient and hopefully more reliable. The new algorithm de-constructs each line after converting it to the system's native line ending. Then it searches for the key and returns value. The rewrite also normalizes line endings to match the system's within the entry itself before returning the final output. This should make things more stable and predictable.
### CSTNet compatibility
For point releases (such as this), QuickFennec.CST will remain under the CSTNet namespace for compatibility reasons. CSTNet will be moved to QuickFennec.CST namespace starting with 1.1.
### Known issues
- Skipping comments is still a little buggy.
- Multiline parsing with the v2 format is still a little unpredictable.
## 1.0.0
- Initial release.

View file

@ -19,7 +19,8 @@
"metadata": {},
"source": [
"using System.IO;\n",
"using System.Text;"
"using System.Collections.Generic;\n",
"using System.Text.RegularExpressions;"
],
"outputs": []
},
@ -30,89 +31,111 @@
"source": [
"public static class CST\n",
"{\n",
" public static string Parse(string cst, int key, params string[] args)\n",
" const char CARET = '^';\n",
" static readonly string _lf = \"\\u000A\";\n",
" static readonly string _cr = \"\\u000D\";\n",
" static readonly string _crlf = \"\\u000D\\u000A\";\n",
" static readonly string _ls = \"\\u2028\";\n",
"\n",
" /// <summary>\n",
" /// Gets the value from the integer-based key.\n",
" /// </summary>\n",
" /// <returns>Returns the entry</returns>\n",
" public static string Parse(string content, int key)\n",
" {\n",
" var entries = NormalizeEntries(cst);\n",
" return GetEntry(entries, $\"{key}\", args);\n",
" var entries = NormalizeEntries(content);\n",
" return GetEntry(entries, key.ToString());\n",
" }\n",
"\n",
" public static string Parse(string cst, string key, params string[] args)\n",
" /// <summary>\n",
" /// Gets the value from the string-based key.\n",
" /// </summary>\n",
" /// <returns>Returns the entry</returns>\n",
" public static string Parse(string content, string key)\n",
" {\n",
" var entries = NormalizeEntries(cst);\n",
" return GetEntry(entries, key, args);\n",
" var entries = NormalizeEntries(content);\n",
" return GetEntry(entries, key);\n",
" }\n",
"\n",
" static IEnumerable<string> NormalizeEntries(string cst)\n",
" /// <summary>\n",
" /// Replaces the document's line endings with the native system line endings.\n",
" /// </summary>\n",
" /// <remarks>This stage ensures there are no crashes during parsing.</remarks>\n",
" static IEnumerable<string> NormalizeEntries(string content)\n",
" {\n",
" var lineBreaks = new string[] { \"^\\u000A\", \"^\\u000D\", \"^\\u000A\" };\n",
"\n",
" foreach (var line in lineBreaks)\n",
" /* \n",
" I tried putting the end carets with the different\n",
" line endings in with the split function but it didn't work \n",
" */\n",
" if (!content.Contains($\"{CARET}{Environment.NewLine}\"))\n",
" {\n",
" var eol = Environment.NewLine; // System's line break\n",
" if (content.Contains($\"{CARET}{_lf}\"))\n",
" content = content.Replace($\"{CARET}{_lf}\",\n",
" $\"{CARET}{Environment.NewLine}\");\n",
"\n",
" // If the new line matches the system's, do nothing\n",
" if (line.Contains(eol))\n",
" if (content.Contains($\"{CARET}{_cr}\"))\n",
" content = content.Replace($\"{CARET}{_cr}\",\n",
" $\"{CARET}{Environment.NewLine}\");\n",
"\n",
" if (content.Contains($\"{CARET}{_crlf}\"))\n",
" content = content.Replace($\"{CARET}{_crlf}\",\n",
" $\"{CARET}{Environment.NewLine}\");\n",
"\n",
" if (content.Contains($\"{CARET}{_ls}\"))\n",
" content = content.Replace($\"{CARET}{_ls}\",\n",
" $\"{CARET}{Environment.NewLine}\");\n",
" }\n",
"\n",
"\n",
" var entries = content.Split(new[] { $\"{CARET}{Environment.NewLine}\" },\n",
" StringSplitOptions.RemoveEmptyEntries);\n",
" var newContent = new List<string>();\n",
"\n",
" foreach (var entry in entries)\n",
" {\n",
" // Skip comments\n",
" if (entry.StartsWith(@\"//\") || entry.StartsWith(\"#\") ||\n",
" entry.StartsWith(\"/*\") || entry.EndsWith(\"*/\"))\n",
" continue;\n",
"\n",
" cst.Replace(line, eol);\n",
" newContent.Add(entry);\n",
" }\n",
"\n",
" return cst.Split(lineBreaks, StringSplitOptions.RemoveEmptyEntries);\n",
"\n",
" return newContent;\n",
" }\n",
"\n",
" static string ArgumentParser(string content, string[] args)\n",
" static string GetEntry(IEnumerable<string> entries, string key)\n",
" {\n",
" var sb = new StringBuilder();\n",
"\n",
" for (var i = 0; i < content.Length; i++)\n",
" {\n",
" var curArgs = content.Substring(i, 1);\n",
" var argsCounter = 0;\n",
"\n",
" if (curArgs.Contains(\"%\"))\n",
" {\n",
" if (argsCounter < args.Length)\n",
" {\n",
" sb.Append(curArgs.Replace(\"%\", args[argsCounter]));\n",
" i++;\n",
" }\n",
" }\n",
" else\n",
" sb.Append(curArgs);\n",
" }\n",
"\n",
" return sb.ToString();\n",
" }\n",
"\n",
" static string GetEntry(IEnumerable<string> entries, string key,\n",
" params string[] args)\n",
" {\n",
" var translation = \"[ENTRY NOT FOUND]\";\n",
"\n",
" // Search through array\n",
" // Search through list\n",
" foreach (var entry in entries)\n",
" {\n",
" // Locate index, trim carets and return translation\n",
" if (!entry.StartsWith(key))\n",
" continue;\n",
" \n",
" const char caret = '^';\n",
"\n",
" var startIndex = entry.IndexOf(caret.ToString(),\n",
" StringComparison.OrdinalIgnoreCase);\n",
"\n",
" var startIndex = entry.IndexOf(CARET);\n",
" var line = entry.Substring(startIndex);\n",
"\n",
" var content = line.TrimStart(caret).TrimEnd(caret);\n",
" if (!line.Contains(Environment.NewLine))\n",
" {\n",
" if (line.Contains(_lf))\n",
" line = line.Replace(_lf, Environment.NewLine);\n",
"\n",
" if (args.Length > 0)\n",
" translation = ArgumentParser(content, args);\n",
" else\n",
" translation = content;\n",
" if (line.Contains(_cr))\n",
" line = line.Replace(_cr, Environment.NewLine);\n",
"\n",
" if (line.Contains(_crlf))\n",
" line = line.Replace(_crlf, Environment.NewLine);\n",
"\n",
" if (line.Contains(_ls))\n",
" line = line.Replace(_ls, Environment.NewLine);\n",
" }\n",
"\n",
" return line.TrimStart(CARET).TrimEnd(CARET);\n",
" }\n",
"\n",
" return translation;\n",
" return \"[ENTRY NOT FOUND]\";\n",
" }\n",
"}"
],
@ -125,16 +148,18 @@
"source": [
"var v1Path = Path.Combine(Environment.CurrentDirectory, \"data\", \"v1.cst\");\n",
"var v1File = File.ReadAllText(v1Path);\n",
"var singleLine = CST.Parse(v1File, 1, \"FENNEC\");\n",
"var multiLine = CST.Parse(v1File, 2);\n",
"Console.WriteLine($\"Single line:{Environment.NewLine}{singleLine}\");\n",
"Console.WriteLine($\"Multiline:{Environment.NewLine}{multiLine}\");"
"var one = CST.Parse(v1File, 1);\n",
"var three = CST.Parse(v1File, 3);\n",
"var four = CST.Parse(v1File, 4);\n",
"Console.WriteLine($\"One:{Environment.NewLine}{one}\");\n",
"Console.WriteLine($\"Three:{Environment.NewLine}{three}\");\n",
"Console.WriteLine($\"Four:{Environment.NewLine}{four}\");"
],
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": "Single line:\r\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Proin ac dictum orci, at tincidunt nulla. Donec aliquet, FENNEC eros non interdum posuere, ipsum sapien molestie nunc, nec facilisis libero ipsum et risus. In sed lorem vel ipsum placerat viverra.\r\n"
"text/plain": "One:\r\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Proin ac dictum orci, at tincidunt nulla. Donec aliquet, %1 eros non interdum posuere, ipsum sapien molestie nunc, nec facilisis libero ipsum et risus. In sed lorem vel ipsum placerat viverra.\r\n"
},
"execution_count": 1,
"metadata": {}
@ -142,7 +167,15 @@
{
"output_type": "execute_result",
"data": {
"text/plain": "Multiline:\r\n[ENTRY NOT FOUND]\r\n"
"text/plain": "Three:\r\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam venenatis ac odio ut pretium. Interdum et malesuada fames ac ante ipsum primis in faucibus. Donec semper turpis tempor, bibendum sapien at, blandit neque. Vivamus hendrerit imperdiet elit, vel sollicitudin nulla luctus vel. Vivamus nisl quam, feugiat a diam aliquam, iaculis vestibulum nunc. Maecenas euismod leo enim, faucibus ultrices ipsum semper eu. Praesent ullamcorper justo at maximus ultricies.\r\n"
},
"execution_count": 1,
"metadata": {}
},
{
"output_type": "execute_result",
"data": {
"text/plain": "Four:\r\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce justo dui, rhoncus a pulvinar sit amet, fermentum vitae lorem. Maecenas nec nisi sit amet eros rutrum congue. In sagittis suscipit arcu, ac vestibulum nunc feugiat volutpat.\r\n\r\nVivamus consequat velit dui, sit amet rhoncus dui malesuada a. Maecenas hendrerit commodo mi et scelerisque. Cras pharetra ultrices aliquam. Praesent ac efficitur magna, vitae scelerisque metus.\r\n"
},
"execution_count": 1,
"metadata": {}
@ -157,7 +190,7 @@
"var v2Path = Path.Combine(Environment.CurrentDirectory, \"data\", \"v2.cst\");\n",
"var v2File = File.ReadAllText(v2Path);\n",
"var singleLineV2 = CST.Parse(v2File, \"Singleline\");\n",
"var multiLineV2 = CST.Parse(v2File, \"Multiline\", \"DOG\", \"CAT\");;\n",
"var multiLineV2 = CST.Parse(v2File, \"Multiline\");\n",
"Console.WriteLine($\"Single line v2:{Environment.NewLine}{singleLineV2}\");\n",
"Console.WriteLine($\"Multiline v2:{Environment.NewLine}{multiLineV2}\");"
],

View file

@ -1,4 +1,6 @@
Singleline^Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed ultricies nulla eu tortor mattis, dictum posuere lacus ornare. Maecenas a massa in ligula finibus luctus eu vitae nibh. Proin imperdiet dapibus mauris quis placerat.^
Multiline ^Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc gravida nunc non justo pretium consectetur. Sed tempus libero ac ligula aliquam elementum. Duis vitae interdum leo. Sed semper nulla %1 a lectus dictum dictum. Ut mattis eu tortor in bibendum. Integer mattis tincidunt aliquet. Vestibulum ante ipsum primis in faucibus orci %2 luctus et ultrices posuere cubilia Curae; Fusce quis orci nisl.
/* this is a
test comment */
Multiline^Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc gravida nunc non justo pretium consectetur. Sed tempus libero ac ligula aliquam elementum. Duis vitae interdum leo. Sed semper nulla %1 a lectus dictum dictum. Ut mattis eu tortor in bibendum. Integer mattis tincidunt aliquet. Vestibulum ante ipsum primis in faucibus orci %2 luctus et ultrices posuere cubilia Curae; Fusce quis orci nisl.
Quisque vehicula, nisi ut scelerisque sodales, nisi ipsum sodales ipsum, in rutrum tellus lacus sed nibh. Etiam mauris velit, elementum sed placerat et, elementum et tellus. Duis vitae elit fermentum, viverra lorem in, lobortis elit. Maecenas eget nibh et lectus auctor dignissim.^