Streaming Large Data Optimally across the internet (C#).
Building modern software applications requires proper planning specifically in data management. The traditional approach of sending data just by calling httpclient will not work all the time. The amount of data to send over TCP will determine the approach of passing data via API.
Slow API calls can lead to APP crashing, App Crawling, bad user experience, high resource consumption, and poor scalability and performance. There are different ways but this is a good one to consider, using stream to improve performance and memory usage.
HTTP operates over TCP, data is sent in packets, each with a maximum size of less than 1 Mb, often relatively small. Any data of any sort that is being communicated via HTTP-based API arrives sequentially in these packets. Sometimes the server can also limit the volume of data to be sent. Sending large records as much as 20 MB or more can be handled efficiently with this approach by passing the common hurdles of large data.
Data can be transmitted and retrieved via the response.Content when the data has arrived via HTTP Client. This is where we do further processing with the data in our application. This is straightforward to work with
The above screenshot can work with transferring small data via the HTTP client. Now in some instances where large data want to be sent, they can stringify. This is an improved way but not the best approach as string can still assume a large memory that can still have bad performance.
Now, let’s look at the crux of the day where you are advised to use this instead as it is efficient and scalable. This involves using a skip the create an in-memory string altogether. A stream represents an abstraction of a sequence of bytes in the form of files, input/output devices, or network traffic. The Stream class in C# is an abstract class that provides methods to transfer bytes. You can reuse this code as an interface for sending data. Let’s check this via the code
We use StreamContent over StringContent to enable the transmission of data in a more streamlined and resource-efficient manner.
From the client side, you can retrieve the data like this
With this approach, you can send large data via HTTP Client without shortchanging it for performance. Other data can be sent across such as sending them in chunks but then it is based on the direction of your application and where data is flowing from.
Be concerned about the performance of your application when transferring data. Stream at your rescue. kindly share with your network.
