Feature/Improved Tokens #7980

chidozieononiwu · 2024-03-29T20:32:02Z

Switch C# APIView Parser from Flat list of Tokens to Tree Tokens

praveenkuttappan · 2024-04-02T16:26:35Z

Can you please include a markdown about token structure and token types so it will be easier for language owners to make the change?

src/dotnet/APIView/APIView/AssemblyInfo.cs

JonathanGiles · 2024-04-15T22:59:04Z

Can you please include a markdown about token structure and token types so it will be easier for language owners to make the change?

Yep - waiting on this so I can add Java support.

src/dotnet/APIView/APIView/APIView.csproj

src/dotnet/APIView/APIView/Model/TokenTreeModel.cs

src/dotnet/APIView/APIView/Model/TreeTokenCodeFile.cs

src/dotnet/APIView/APIView/Model/TokenTreeModel.cs

tjprescott

Please beef up the documentation.

tools/apiview/parsers/CONTRIBUTING.md

tools/apiview/parsers/csharp-api-parser/Program.cs

praveenkuttappan

Parser name needs to be modified along with other small changes that can improve perf.

src/dotnet/APIView/APIView/Model/CodeDiagnostic.cs

src/dotnet/APIView/APIView/Model/TokenTreeModel.cs

tools/apiview/parsers/csharp-api-parser/CSharpAPIParser.csproj

tools/apiview/parsers/CONTRIBUTING.md

src/dotnet/APIView/APIView/Model/TokenTreeModel.cs

christothes · 2024-06-10T22:00:28Z

tools/apiview/parsers/csharp-api-parser/Program.cs

+return rootCommand.InvokeAsync(args).Result;
+
+
+static void HandlePackageFileParsing(Stream stream, FileInfo packageFilePath, DirectoryInfo OutputDirectory, string outputFileName, bool runAnalysis)


This can potentially be simplified with the NuGet.Protocol package - samples here

src/dotnet/APIView/APIView/Model/TokenTreeModel.cs

maririos · 2024-06-18T13:48:24Z

tools/apiview/parsers/CONTRIBUTING.md

+    Url
+  ```
+
+### APITreeNode


Documenting here the classes won't scale and the documentation will get out of date. Instead, let's do the documentation in the class itself and point people there.

Personally, I would rather not try to interpret C# code comments when working out how to write my parser in Java - I only fall back to looking there when the markdown is missing some details.

100% agree with @JonathanGiles. Parser authors should not be forced to dig around in C# source code as a substitute for actual documentation.

tools/apiview/parsers/CONTRIBUTING.md

src/dotnet/APIView/APIView/Model/TokenTreeModel.cs

src/dotnet/APIView/APIView/CodeFileTokensBuilder.cs

src/dotnet/APIView/APIView/Analysis/Analyzer.cs

src/dotnet/APIView/APIView/Model/CodeDiagnostic.cs

src/dotnet/APIView/APIView/Model/TokenTreeModel.cs

Parse C# Assemblly to Token Tree Add Token Tree Documentation Add LineDefinitionId and HideFromNavigation Undo internals visible to setting Sort NameSpaces and Types Ensure Each node has Name and Id Serialize to Message Pack Add Tags to StructuredToken update tree Token Parser Documentation Attempt to Improve Serialized JSON Size Properties for Serialization and Deserialization Use shorter names for diagnostics update APIView Parser Contributing guide Delete TreeTokenCodeFile Update APIView C# Parser Project Name Add MessagePack Serialization Switch to System.Json.Text working Update Contributing Doc Add Navigate to ID Make Parser a dotnet Tool Update Parser Version for CodeFile App Parser Style Property Output GZipped TokenFile Switch to Human Readable Json Property Names, use compression Fix some issues raised in pull request Updating Contributing.md Update Contributing Docs

Parse C# Assemblly to Token Tree Add Token Tree Documentation Add LineDefinitionId and HideFromNavigation Undo internals visible to setting Sort NameSpaces and Types Ensure Each node has Name and Id Serialize to Message Pack Add Tags to StructuredToken update tree Token Parser Documentation Attempt to Improve Serialized JSON Size Properties for Serialization and Deserialization Use shorter names for diagnostics update APIView Parser Contributing guide Delete TreeTokenCodeFile Update APIView C# Parser Project Name Add MessagePack Serialization Switch to System.Json.Text working Update Contributing Doc Add Navigate to ID Make Parser a dotnet Tool Update Parser Version for CodeFile App Parser Style Property Output GZipped TokenFile Switch to Human Readable Json Property Names, use compression Fix some issues raised in pull request Updating Contributing.md Update Contributing Docs initial pass to documentation fix pr feedback one missed

Initial pass to improve token documentation

chidozieononiwu requested a review from a team as a code owner March 29, 2024 20:32

chidozieononiwu requested review from maririos and praveenkuttappan and removed request for a team March 29, 2024 20:33

weshaggard reviewed Apr 3, 2024

View reviewed changes

src/dotnet/APIView/APIView/AssemblyInfo.cs Outdated Show resolved Hide resolved

chidozieononiwu force-pushed the feature/ImprovedTokens branch 2 times, most recently from a935767 to dd99fc0 Compare April 17, 2024 22:27

chidozieononiwu force-pushed the feature/ImprovedTokens branch from dd99fc0 to 512bcb3 Compare May 6, 2024 23:30

chidozieononiwu force-pushed the feature/ImprovedTokens branch 2 times, most recently from 9d2a1fb to c663a95 Compare May 14, 2024 21:18

chidozieononiwu force-pushed the feature/ImprovedTokens branch 2 times, most recently from 23bb386 to 3c3723e Compare May 31, 2024 15:56