I’ve recently been trying out the new System.Text.Json JSON Parser that is now built into .NET Core 3+, replacing NewtonSoft.Json (Sometimes called JSON.NET) as the default JSON parser in ASP.NET Core. There has been quite a few gotchas and differences between the two libraries, but none more interesting than the following piece of documentation :
During deserialization, Newtonsoft.Json does case-insensitive property name matching by default. The System.Text.Json default is case-sensitive, which gives better performance since it’s doing an exact match.
I found this interesting, especially the last line which suggests that doing exact matches by default results in much better performance.
Interestingly enough, there is also the following piece of info :
If you’re using System.Text.Json indirectly by using ASP.NET Core, you don’t need to do anything to get behavior like Newtonsoft.Json. ASP.NET Core specifies the settings for camel-casing property names and case-insensitive matching when it uses System.Text.Json.
So that essentially means when we switch to System.Text.Json in an ASP.NET Core project specifically, things will be case insensitive by default, and by extension, have slightly worse performance than forcing things to be case sensitive. By the way, for the record, I think this should be the default behaviour because in 99% of Web SPA cases, the front end will be using javascript which typically is written using camelCase, and if the backend is in C#, the properties are typically written in PascalCase.
But here’s the thing I wondered. When everyone’s out there posting benchmarks, are they benchmarking case sensitive or case insensensitive JSON parsing? In another blog post I and many others often refer back to for JSON benchmarks (https://michaelscodingspot.com/the-battle-of-c-to-json-serializers-in-net-core-3/) they are actually doing the case sensitive parsing which again, I don’t think is going to be the reality for the majority of use cases.
Let’s dig a little more!
Benchmarking
The first thing I wanted to do was create a simple benchmark to test my hypothesis. That is, does deserializing data with case insensitivity turned on slow down the deserialization process.
I’m going to use BenchmarkDotNet for this purpose. The class I want to deserialize looks like so :
public class MyClass
{
public int MyInteger { get; set; }
public string MyString { get; set; }
public List<string> MyList { get; set; }
}Now I also had an inkling of a theory this wouldn’t be as simple as first thought. I had a hunch that possibly that even when case insensitivity was turned on, if it could find an exact match first, it would attempt to use that anyway. e.g. It’s possible that the code would look something like this pseudo code :
if(propertyName == jsonProperty)
{
//We found the property on an exact match.
}else if(propertyName.ToLower() == jsonProperty.ToLower())
{
//We found it after ToLowering everything.
}With that in mind, I wanted my benchmark to test serializing both PascalCase names (So exact match), and camelCase names. My benchmark looked like so :
public class SystemTextVsJson
{
private readonly JsonSerializerOptions options = new JsonSerializerOptions() { PropertyNameCaseInsensitive = true };
private const string _jsonStringPascalCase = "{\"MyString\" : \"abc\", \"MyInteger\" : 123, \"MyList\" : [\"abc\", \"123\"]}";
private const string _jsonStringCamelCase = "{\"myString\" : \"abc\", \"myInteger\" : 123, \"myList\" : [\"abc\", \"123\"]}";
[Benchmark]
public MyClass SystemTextCaseSensitive_Pascal()
{
return JsonSerializer.Deserialize<MyClass>(_jsonStringPascalCase);
}
[Benchmark]
public MyClass SystemTextCaseInsensitive_Pascal()
{
return JsonSerializer.Deserialize<MyClass>(_jsonStringPascalCase, options);
}
[Benchmark]
public MyClass SystemTextCaseSensitive_Camel()
{
return JsonSerializer.Deserialize<MyClass>(_jsonStringCamelCase);
}
[Benchmark]
public MyClass SystemTextCaseInsensitive_Camel()
{
return JsonSerializer.Deserialize<MyClass>(_jsonStringCamelCase, options);
}
}And just so we are all on the same page, the machine I’m running this on looks like :
BenchmarkDotNet=v0.12.0, OS=Windows 10.0.18362 AMD Ryzen 7 2700X, 1 CPU, 16 logical and 8 physical cores .NET Core SDK=3.1.100
Now onto our results :
| Method | Mean | Error | StdDev |
|---|---|---|---|
| SystemTextCaseSensitive_Pascal | 1.511 us | 0.0298 us | 0.0279 us |
| SystemTextCaseInsensitive_Pascal | 1.538 us | 0.0052 us | 0.0049 us |
| SystemTextCaseSensitive_Camel | 1.877 us | 0.0297 us | 0.0277 us |
| SystemTextCaseInsensitive_Camel | 2.548 us | 0.0164 us | 0.0145 us |
Interesting. Very interesting. So a couple of things that stick out to me immediately.
If we are using PascalCase for our property names on both ends (in the JSON and our C# class), then the case sensitivity setting doesn’t matter all too much. This proves my initial thoughts that it may try for an exact match no matter the setting as that’s likely to be faster than any string manipulation technique.
Next. Slightly of interest is that when parsing with case sensitivity turned on, when there is no match (e.g. You have screwed up the casing on one of the ends), it runs slightly slower. Not by much. But enough to be seen in the results. This is probably because it tries to do some extra “matching” if it can’t find the exact match.
Finally. Oof. Just as we thought. When we are doing case insensitive matching and our incoming data is camelCase with the class being PascalCase, the benchmark is substantially slower than exact matching. And I just want to remind you, the default for ASP.NET Core applications is case insensitive.
So, how does this actually stack up?
Benchmarking Against Newtonsoft
The interesting thing here was if we are comparing apples to apples, Newtonsoft also does case insensitive matching but it does so by default. So when we do any benchmarking against it, we should try and do so using similar settings if those settings would be considered the norm.
With that in mind, let’s do this benchmark here :
public class SystemTextVsJson
{
private readonly JsonSerializerOptions options = new JsonSerializerOptions() { PropertyNameCaseInsensitive = true };
private const string _jsonStringCamelCase = "{\"myString\" : \"abc\", \"myInteger\" : 123, \"myList\" : [\"abc\", \"123\"]}";
[Benchmark]
public MyClass SystemTextCaseInsensitive_Camel()
{
return JsonSerializer.Deserialize<MyClass>(_jsonStringCamelCase, options);
}
[Benchmark]
public MyClass NewtonSoftJson_Camel()
{
return Newtonsoft.Json.JsonConvert.DeserializeObject<MyClass>(_jsonStringCamelCase);
}
}And the results?
| Method | Mean | Error | StdDev |
|---|---|---|---|
| SystemTextCaseInsensitive_Camel | 2.555 us | 0.0106 us | 0.0099 us |
| NewtonSoftJson_Camel | 2.852 us | 0.0104 us | 0.0087 us |
Much much much closer. So now that we are comparing things on an even footing the performance of System.Text.Json to NewtonSoft actually isn’t that much better in terms of raw speed. But what about memory? I hear that touted a lot with the new parser.
Memory Footprint
BenchmarkDotNet gives us the ability to also profile memory. For our test, I’m going to keep the same benchmarking class but just add the MemoryDiagnoser attribute onto it.
[MemoryDiagnoser]
public class SystemTextVsJson
{
}And the results
| Method | Allocated |
|---|---|
| SystemTextCaseInsensitive_Camel | 408 B |
| NewtonSoftJson_Camel | 3104 B |
Wow, credit where credit is due, that’s a very impressive drop. Now again I’m only testing with a very minimal JSON string, but I’m just looking to do a comparison between the two anyway.
Final Thoughts
Why did I make this post in the first place? Was it to crap all over System.Text.Json and be team JSON.NET all the way? Not at all. But I have to admit, there is some level of frustration when moving to using System.Text.Json when it doesn’t have the “features” that you are used to in JSON.NET, but it’s touted as being much faster. Then when you dig a little more in the majority of use cases (case insensitive), it’s not actually that much faster.
And I have to point out as well. That literally everytime I’ve written a benchmarking post for C# code, I’ve managed to get something wrong where I didn’t know the compiler would optimize things out etc and someone jumps in the Reddit comments to call me an idiot (Will probably happen with this one too! Feel free to drop a comment below!). So you can’t really blame people doing benchmarks across JSON parsers without realizing the implications of casing because the fact ASP.NET Core has specific defaults that hide away this fact means that you are unlikely to run into the issue all that often.
If you are in the same boat as me and trying to make the leap to System.Text.Json (Just so you stay up to date with what’s going on), I have a post sitting in my drafts around gotchas with the move. Case sensitivity is a big one but also a bunch of stuff on various defaults, custom converters, null handling etc which were all so great in JSON.NET and maybe a little less great in System.Text.Json. So watch this space!
Although the post of Michael has got some attention I think my more complete coverage of serializers, which he also mentions, is still the benchmark: https://aloiskraus.wordpress.com/2019/09/29/net-serialization-benchmark-2019-roundup/.
Your test looks fine to me for your specific point. As far as I have read the author of Json.NET which is now with MS had some rough feedback regarding feature set and performance of the .NET Core version. Only using Span wont make things much faster.
I have noticed you have an AMD CPU. Not sure how much different the timings compared to Haswell or later CPU generations the single core performance is there. But a comparison of Intel vs AMD could be interesting as well.
There is a big difference with certain amd CPUs due to not having the later extentions
Thanks for the post and the information. Two points/questions
1. You mention being concerned with the number of real-world use cases but then use a very small JSON string. I wouldn’t expect a huge multi-megabyte string but something with a couple of objects and several values in the list might be more ‘real-world’
2. Interesting to know if using the Property attributes to decorate the methods and then using a case sensitive approach would have an effect on timing (eg [JsonPropertyName(“myInteger “)])
Regarding question 2, using `JsonPropertyName` with case sensitive gets you back to the 2x performance. I have provided the details in a response below.
Note: I am a member of the .NET Core team at Microsoft and work on the System.Text.Json library.
Doing an apples to apples comparison with microbenchmarks can definitely be helpful, but it is also important to keep actual user scenarios in mind and see how the end-to-end is impacted.
The buit-in JSON serializer is intentionally designed with certain performance and security principles, while giving the user control/flexibility (pay for play). So, it would be helpful to benchmark code that people should and would actually end up writing against the respective libraries. For example, System.Text.Json is optimized for UTF-8 data and by benchmarking UTF-16 strings (because that is what Newtonsoft.Json provides), with the hopes of doing comparisons on an equal fotting doesn’t take the whole picture into account. Let’s say you are reading from a file on disk (which is saved as UTF-8 encoded text). Transcoding it to a UTF-16 string first, and then processing the JSON is unnecessary overhead. Why not pass in the UTF-8 bytes directly to the JSON serializer and save the cost of transcoding bakc and forth?
It is expected for folks (at least in some cases) to be aware of these differences and potentially leverage then to get the best performance, if it is important for their scenario. A drop-in replacement is still faster and allocates significantly less as you observed, but yes the results might vary. In fact, you could come up with all sorts of .NET models (based on how they are structured, what collection properties are used, etc.), that can show different performance characteristics. It is difficult to create a single representative benchmark to draw conclusions from. Using a sample benchmark and its results should be caveated with “your milage may vary” and “please measure your specific scenario, yourself!”. So, the question should be more around what performance are you capable off with the new library.
There are certainly scenarios where folks are looking for an exact match (either because their JSON and POCO actually match, or they attribute their POCO with the matching names they expect, in whatever casing). The System.Text.Json library provides that option and opt’d to make that the default for a few reasons, one of which being performance (also avoiding any “magic”/guess work on behalf of the user by making the intention explicit, which tends to be easier for developers to reason about). We were able to make certain optimizations that are only feasible when doing an exact match (including some name caching), that can’t be done with case insensitive comparisons. Therefore, it isn’t suprising to see the performance difference between those two options in isolation (and is certainly expected). You could also consider using the camel case naming policy rather than opting for case insensitive comparisons (why allow “myInTEgEr” as the property name when you only need “myInteger”). Like you mentioned, many web SPAs use camel case. That is why the aspnet defaults have JsonNamingPolicy set to camel case (along with allowing case insensitive comparison). For that scenario, with camel cased payloads, you still get the 2x performance.
To summarize, we have at least the following factors to consider (see the benchmarks below to get the whole picture):
– Different models/paylod sizes and user scenarios
– Annotate your model with JsonPropertyName attribute for exact match
– Use JsonNamingPolicy instead of (or in addition to) case insensitive string comparison
– UTF-8 JSON data vs UTF-16
All that said, we are actively working on improving performance even more for the 5.0 release, including finding cases where there was unnecessary overhead or some perf gap went unnoticed. The great thing about .NET Core being open source is you can see how it works. If you have some interesting or cool ideas on how to make those scenarios faster, that would be awesome!
I also appreciate blogs like these which encourage diving deeper, doing your own benchmarking, and understanding the nuances 🙂 Thank you.
As an aside: The `SystemTextCaseSensitive_Camel` benchmark is returning an empty `MyClass` object because none of the JSON properties map to the properties on the POCO. I don’t know what is the intent of that benchmark.
P.S. I am looking forward to your upcoming articles and interested in your findings. Feel free to leverage some of the existing docs on differences in default behaviors:
https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-migrate-from-newtonsoft-how-to#differences-in-default-jsonserializer-behavior-compared-to-newtonsoftjson
Hey there,
Thanks for the reply and your own benchmarks. The CamelCasing naming property is probably the most interesting thing to me because I think that’s the most realistic “fix” if you want to call it that. Annotations a little less so because having to annotate 100% of your model is a bit overkill (I think personally).
>The SystemTextCaseSensitive_Camel benchmark is returning an empty MyClass object because none of the JSON properties map to the properties on the POCO. I don’t know what is the intent of that benchmark.
That relates to this part “I had a hunch that possibly that even when case insensitivity was turned on, if it could find an exact match first, it would attempt to use that anyway”. I kept it there to see if there was some sort of performance hit when given a model that *doesn’t* match in anyway. To see if it would do things like checking for the exact match, and then maybe doing something else to try and match it.
So from the results we saw that if we have a parser set to case sensitive, and we give it a PascalCase vs camelCase input, that the camelCase input is slower (And yes, the model ends up empty).
> The CamelCasing naming property is probably the most interesting thing to me because I think that’s the most realistic “fix” if you want to call it that.
Agreed. Within the context of aspnet apps (like mvc), that “fix” is already set as the default option (camel case naming policy + case insensitive comparison), and when both are set, performance is decent when the payload is camel case.
https://github.com/dotnet/aspnetcore/blob/2481862682c6f503f3f5e65a38369f15817487f1/src/Mvc/Mvc.Core/src/JsonOptions.cs#L9-L30
> Annotations a little less so because having to annotate 100% of your model is a bit overkill (I think personally).
Yep, that’s totally fair.
Btw, for reference, here is the gist and the benchmark results (in a more readable format):
https://gist.github.com/ahsonkhan/8f86114245fd15b46d084ddfc4935d41
https://gist.github.com/ahsonkhan/8f86114245fd15b46d084ddfc4935d41#gistcomment-3157391
It seems the defaults for ASP.NET Core 3.1 have changed. JsonSerializerOptions.PropertyNameCaseInsensitive is now set by default to be false, which I thought would yield better results, than having JsonSerializerOptions.PropertyNameCaseInsensitive = true (and JsonSerializerOptions.PropertyNamingPolicy = JsonNamingPolicy.CamelCase)
Though, why was this changed again? Is it indeed to improve performance even more, as comparisons in case-sensitivity is better than case-insensitivity, or other reasons?
As someone who’s trying to decide whether the cost of moving to System.Json.Text is worthwhile, I’m more interested in what the typical time taken to process a web endpoint request from woe-to-go is rather than just the de-serialization/serialization part. I.e. assuming a typical endpoint might take 250ms from the time the client connects to the time the last byte of the response is written, how many ms am I likely to save switching serializers? Obviously this depends significantly on the *size* of the request and response (certainly it’s not uncommon with very large responses for them take the bulk of the processing time to be serialized and sent back over the socket), but if it turns out even with a 100,000 byte response that we’d only save 10ms per call, I’m not convinced it’s worth making the move.
Actually I did a bit more reading and it does seem switching to System.Text.Json significantly improves ASP.NET performance, at least partly because it works directly on the UTF8 bytes from the socket layer.
https://michaelscodingspot.com/the-battle-of-c-to-json-serializers-in-net-core-3/
Having said that, the benchmarks we’ve done internally of our own APIs isn’t showing the actual time converting the binary representation to JSON to be the time-consuming part, it’s mostly writing the data to the socket.
Most of the performance benefits come when you do everything in Memory / Span and skip going through C# strings and the UTF16 conversion and memory allocation. eg your web stack gives you a pointer into the received message buffer. Kind of hard to test with small benchmarks.
If you go through strings your bench mark will miss a significant part of the benefit.
Note: I am a member of the .NET Core team at Microsoft and work on the System.Text.Json library.
If you re-run the benchmarks on a recent 5.0 build you will find the case-insensitive and missing-property scenarios as fast (or nearly as fast) as the default case-sensitive benchmark. This was addressed in https://github.com/dotnet/runtime/pull/35848.
I’m going to call you an idiot to fulfil the quota :p, although I really don’t see anything idiotic here. In contrast, this is a very important article because as you mentioned, in the real web world, you will almost always be using JSON serialization from within ASP.NET Core… which has the different, slower, Newtonsoft.Json-compatible JsonSerializer settings enabled. As you mentioned, almost none of the benchmarks touting the speed of System.Text.Json over Newtonsoft.Json use these same settings, which makes most of those benchmarks functionally useless.
Having to apply those different settings also makes working with System.Text.Json extremely aggravating, but that’s an entirely different topic.
Thank you for your enlightening article.
I’m sticking with Newtonsoft. I see no need to move to MS’s alternative.