Skip to content
This repository was archived by the owner on Jan 23, 2023. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
a19d268
Add an in-box array-backed IBufferWriter<T>
ahsonkhan Apr 15, 2019
b522e0c
Update Utf8JsonWriter ref and main writer file.
ahsonkhan Apr 15, 2019
341d937
Fix JsonWriter WriteValue APIs.
ahsonkhan Apr 15, 2019
9caf27c
Use Environment.NewLine static and invert some stream conditions.
ahsonkhan Apr 15, 2019
38a3834
Update JsonWriter properties and fix serializer build breaks.
ahsonkhan Apr 16, 2019
61f51c4
Update JsonWriter unit tests.
ahsonkhan Apr 16, 2019
1406858
Add xml comments, clean up, and improve test coverage.
ahsonkhan Apr 17, 2019
449d935
Update JsonDocument and JsonSerializer to react to JsonWriter changes.
ahsonkhan Apr 17, 2019
ec4d03f
Normalize the reference assembly.
ahsonkhan Apr 17, 2019
5a060d1
Do not escape/validate comments and update issue number.
ahsonkhan Apr 17, 2019
fc5c82a
Do not escape comments and validate they don't contain embedded
ahsonkhan Apr 17, 2019
b44b6b8
Merge branch 'master' of https://github.com/dotnet/corefx into Redesi…
ahsonkhan Apr 17, 2019
d6213e3
Remove dead code and update issue number in comments.
ahsonkhan Apr 17, 2019
50e832c
Throw InvalidOEx instead of ArgEx if IBW doesn't give requested memory.
ahsonkhan Apr 17, 2019
f94ca69
Fix test build breaks for netfx.
ahsonkhan Apr 17, 2019
6aa5efc
Remove dead code and fix source packaging.
ahsonkhan Apr 18, 2019
5af4802
Merge branch 'master' of https://github.com/dotnet/corefx into Redesi…
ahsonkhan Apr 18, 2019
695c9df
Address PR feedback.
ahsonkhan Apr 18, 2019
4cecd6a
Disable writing floats test on windows
ahsonkhan Apr 18, 2019
4d7a90e
8 digit floats don't work well on older TFMs. Reduce to 7.
ahsonkhan Apr 18, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 187 additions & 0 deletions src/Common/src/System/Buffers/ArrayBufferWriter.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Diagnostics;

namespace System.Buffers
{
/// <summary>
/// Represents a heap-based, array-backed output sink into which <typeparam name="T"/> data can be written.
/// </summary>
#if USE_ABW_INTERNALLY
internal
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this pattern acceptable to use? If not, I am open to suggestions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do this in other places as well, but I think it would be better to invert the condition, such that the type is only public if you explicitly set a compilation constant... otherwise, if a project includes this file, by default it's going to be exposing a new type publicly.

#else
public
#endif
sealed class ArrayBufferWriter<T> : IBufferWriter<T>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add where T:Struct to avoid mixing objects and memory.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to match the IBufferWriter<T> constraint here (i.e. none). If there is strong opposition, we should discuss both outside of this PR in an API review.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather not artificially constrain it, unless there's good reason to.

Copy link
Contributor

@steveharter steveharter Apr 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling GetMemory() would be a bit odd with an array of objects. I'm assuming we don't do anything unsafe with these object pointers like mem copying (in the middle of a gc).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling GetMemory() would be a bit odd with an array of objects.

Why?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scenario is a bit edge case IMO - when would you pass a writeable buffer of objects? There's an indirection in memory (element->object) and "ownership" now becomes a bit more ambiguous - i.e. what if the objects implement IDispose -- who's going to call that?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's no different than storing objects into an array; when do you pass arrays of things, like strings? You also can have the exact same ownership question with structs, which might themselves have resources that have lifetime, or wrap any number of reference types. Limiting it is just an artificial restriction.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that limiting isn't a good thing if we don't need to.

Since Memory has lifetime semantics (don't hold onto it after your method returns etc) it is not the same as an array.

Consider a buffer type that is only for data (or primitives like char or byte - not objects) and in that case there could be some memory clear and copy-like efficiencies regarding clearing after use and shuffling unprocessed elements to the beginning when making room for more data. Originally I was thinking we may be doing some of these operations here which would make it unsafe for objects...

{
private T[] _buffer;
private int _index;

private const int MinimumBufferSize = 256;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't actually MinimumBufferSize, at least not how it's currently used. A better name for the current usage would be DefaultInitialBufferSize, right?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was thinking the same thing. The only concern is that I use this to define the minimum size GetMemory/GetSpan can return as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only concern is that I use this to define the minimum size GetMemory/GetSpan can return as well.

It isn't the minimum, though. It's just the value that you pretend sizeHint was if it's 0. (GetSpan(1) is allowed to return a span of Length 1)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough. I meant minimum in the "default" case, but you brought up a useful clarification.


/// <summary>
/// Creates an instance of an <see cref="ArrayBufferWriter{T}"/>, in which data can be written to,
/// with the default initial capacity.
/// </summary>
public ArrayBufferWriter()
{
_buffer = new T[MinimumBufferSize];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why wouldn't we start with Array.Empty<T>()? Then if it's not actually written to, we don't allocate, and if it is written to, the first allocation will be based on the supplied sizeHint. If a typical usage is creating one of these and then passing it off to something that uses a sizeHint larger than 256, this is going to unnecessarily allocate a throw-away 256-byte buffer.

_index = 0;
}

/// <summary>
/// Creates an instance of an <see cref="ArrayBufferWriter{T}"/>, in which data can be written to,
/// with an initial capacity specified.
/// </summary>
/// <param name="initialCapacity">The minimum capacity with which to initialize the underlying buffer.</param>
/// <exception cref="ArgumentException">
/// Thrown when <paramref name="initialCapacity"/> is not positive (i.e. less than or equal to 0).
/// </exception>
public ArrayBufferWriter(int initialCapacity)
{
if (initialCapacity <= 0)
throw new ArgumentException(nameof(initialCapacity));

_buffer = new T[initialCapacity];
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to bound initialCapacity to MinimumBufferSize from the bottom or is this fine?

_index = 0;
}

/// <summary>
/// Returns the data written to the underlying buffer so far, as a <see cref="ReadOnlyMemory{T}"/>.
/// </summary>
public ReadOnlyMemory<T> WrittenMemory => _buffer.AsMemory(0, _index);

/// <summary>
/// Returns the data written to the underlying buffer so far, as a <see cref="ReadOnlySpan{T}"/>.
/// </summary>
public ReadOnlySpan<T> WrittenSpan => _buffer.AsSpan(0, _index);

/// <summary>
/// Returns the amount of data written to the underlying buffer so far.
/// </summary>
public int WrittenCount => _index;

/// <summary>
/// Returns the total amount of space within the underlying buffer.
/// </summary>
public int Capacity => _buffer.Length;

/// <summary>
/// Returns the amount of space available that can still be written into without forcing the underlying buffer to grow.
/// </summary>
public int FreeCapacity => _buffer.Length - _index;
Copy link
Member

@stephentoub stephentoub Apr 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this? Someone can just do Capacity - WrittenCount.

Also, WrittenCount is an uncommon name. How about Position?

Copy link
Author

@ahsonkhan ahsonkhan Apr 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The serializer uses this but you are right it is primarily a convenience thing. Do you think its exposing too much or do you just think it is unnecessary and not worth the benefit it provides?

Regarding the naming/API changes like this, how do you feel about discussing it all up in a follow up review (post preview 5)? I prefer not to make these changes in PR (in isolation). WrittenCount fits nicer with WrittenMemory/Span (especially when passed in as "offset/count" to stream.write), imo, but I am not super attached to the name.


/// <summary>
/// Clears the data written to the underlying buffer.
/// </summary>
/// <remarks>
/// You must clear the <see cref="ArrayBufferWriter{T}"/> before trying to re-use it.
/// </remarks>
public void Clear()
{
Debug.Assert(_buffer.Length >= _index);
_buffer.AsSpan(0, _index).Clear();
_index = 0;
}

/// <summary>
/// Notifies <see cref="IBufferWriter{T}"/> that <paramref name="count"/> amount of data was written to the output <see cref="Span{T}"/>/<see cref="Memory{T}"/>
/// </summary>
/// <exception cref="ArgumentException">
/// Thrown when <paramref name="count"/> is negative.
/// </exception>
/// <exception cref="InvalidOperationException">
/// Thrown when attempting to advance past the end of the underlying buffer.
/// </exception>
/// <remarks>
/// You must request a new buffer after calling Advance to continue writing more data and cannot write to a previously acquired buffer.
/// </remarks>
public void Advance(int count)
{
if (count < 0)
throw new ArgumentException(nameof(count));

if (_index > _buffer.Length - count)
ThrowInvalidOperationException(_buffer.Length);

_index += count;
}

/// <summary>
/// Returns a <see cref="Memory{T}"/> to write to that is at least the requested length (specified by <paramref name="sizeHint"/>).
/// If no <paramref name="sizeHint"/> is provided (or it's equal to <code>0</code>), some non-empty buffer is returned.
/// </summary>
/// <exception cref="ArgumentException">
/// Thrown when <paramref name="sizeHint"/> is negative.
/// </exception>
/// <remarks>
/// This will never return an empty <see cref="Memory{T}"/> but it can throw
/// if the requested buffer size is not available.
/// </remarks>
/// <remarks>
/// There is no guarantee that successive calls will return the same buffer or the same-sized buffer.
/// </remarks>
/// <remarks>
/// You must request a new buffer after calling Advance to continue writing more data and cannot write to a previously acquired buffer.
/// </remarks>
public Memory<T> GetMemory(int sizeHint = 0)
{
CheckAndResizeBuffer(sizeHint);
Debug.Assert(_buffer.Length >= _index);
return _buffer.AsMemory(_index);
}

/// <summary>
/// Returns a <see cref="Span{T}"/> to write to that is at least the requested length (specified by <paramref name="sizeHint"/>).
/// If no <paramref name="sizeHint"/> is provided (or it's equal to <code>0</code>), some non-empty buffer is returned.
/// </summary>
/// <exception cref="ArgumentException">
/// Thrown when <paramref name="sizeHint"/> is negative.
/// </exception>
/// <remarks>
/// This will never return an empty <see cref="Span{T}"/> but it can throw
/// if the requested buffer size is not available.
/// </remarks>
/// <remarks>
/// There is no guarantee that successive calls will return the same buffer or the same-sized buffer.
/// </remarks>
/// <remarks>
/// You must request a new buffer after calling Advance to continue writing more data and cannot write to a previously acquired buffer.
/// </remarks>
public Span<T> GetSpan(int sizeHint = 0)
{
CheckAndResizeBuffer(sizeHint);
Debug.Assert(_buffer.Length >= _index);
return _buffer.AsSpan(_index);
}

private void CheckAndResizeBuffer(int sizeHint)
{
if (sizeHint < 0)
throw new ArgumentException(nameof(sizeHint));

if (sizeHint == 0)
{
sizeHint = MinimumBufferSize;
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any feedback/concerns on this growth strategy? Any call to GetSpan/GetMemory will return at least 256 bytes, if sizeHint is not specified.

This has consequences when used with an ArrayBufferWriter ctor that has initialCapacity.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was contemplating this when looking earlier. I'd say that sizeHint of 0 should imply sizeHint of 1 (simply "not empty" but avoids growing if it can)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had considered this, but in practice, most workloads would suffer from this since they can't make forward progress with 1 byte. It effectively makes calling GetSpan without an argument useless. I am not sure returning 1 is better. If there are no strong opinions here, I would let scenario-driven user feedback guide the change, and leave it as is for now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in practice, most workloads would suffer from this since they can't make forward progress with 1 byte. It effectively makes calling GetSpan without an argument useless.

I dunno, once the regrow happens it'll double in size, so you'll end up with one call between 1 and 256, then the "oh, I need more than that..." would get a bigger buffer.

I would let scenario-driven user feedback guide the change, and leave it as is for now.

Also seems valid, but if the caller really doesn't have any notion of a hint then it seems like any size is acceptable. Otherwise passing in min/max/accurate seems like the best bet.

}

if (sizeHint > FreeCapacity)
{
int growBy = Math.Max(sizeHint, _buffer.Length);

int newSize = checked(_buffer.Length + growBy);

Array.Resize(ref _buffer, newSize);
}

Debug.Assert(FreeCapacity > 0 && FreeCapacity >= sizeHint);
}

private static void ThrowInvalidOperationException(int capacity)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Seems like this should be named ThrowAdvancedTooFarException, or something like that.

{
throw new InvalidOperationException(SR.Format(SR.BufferWriterAdvancedTooFar, capacity));
}
}
}
14 changes: 14 additions & 0 deletions src/System.Memory/ref/System.Memory.cs
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,20 @@ public static void Reverse<T>(this System.Span<T> span) { }
}
namespace System.Buffers
{
public sealed partial class ArrayBufferWriter<T> : System.Buffers.IBufferWriter<T>
{
public ArrayBufferWriter() { }
public ArrayBufferWriter(int initialCapacity) { }
public int Capacity { get { throw null; } }
public int FreeCapacity { get { throw null; } }
public int WrittenCount { get { throw null; } }
public System.ReadOnlyMemory<T> WrittenMemory { get { throw null; } }
public System.ReadOnlySpan<T> WrittenSpan { get { throw null; } }
public void Advance(int count) { }
public void Clear() { }
public System.Memory<T> GetMemory(int sizeHint = 0) { throw null; }
public System.Span<T> GetSpan(int sizeHint = 0) { throw null; }
}
public static partial class BuffersExtensions
{
public static void CopyTo<T>(this in System.Buffers.ReadOnlySequence<T> source, System.Span<T> destination) { }
Expand Down
3 changes: 3 additions & 0 deletions src/System.Memory/src/Resources/Strings.resx
Original file line number Diff line number Diff line change
Expand Up @@ -143,4 +143,7 @@
<data name="UnexpectedSegmentType" xml:space="preserve">
<value>Unexpected segment type.</value>
</data>
<data name="BufferWriterAdvancedTooFar" xml:space="preserve">
<value>Cannot advance past the end of the buffer, which has a size of {0}.</value>
</data>
</root>
3 changes: 3 additions & 0 deletions src/System.Memory/src/System.Memory.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@
<Compile Include="$(CommonPath)\CoreLib\System\Numerics\Hashing\HashHelpers.cs">
<Link>Common\System\Collections\HashHelpers.cs</Link>
</Compile>
<Compile Include="$(CommonPath)\System\Buffers\ArrayBufferWriter.cs">
<Link>Common\System\Buffers\ArrayBufferWriter.cs</Link>
</Compile>
</ItemGroup>
<ItemGroup>
<ReferenceFromRuntime Include="System.Private.CoreLib" />
Expand Down
Loading