Jsoup Not Fetching Full Page Data? Fix the maxBodySize Limit Issue

Recently, I encountered an issue in our Jsoup-based automation project where the full page data was not being retrieved.

Document doc = Jsoup.connect(performanceTabURL).get(); 

When I printed the doc object, I noticed that the response did not contain the complete page content.

What made this issue more interesting was that it was environment-specific. On one machine/network (client environment), the response was truncated, while on another network, the same code was able to fetch the complete data without any issues.

Although I couldn’t pinpoint the exact root cause for this inconsistent behavior, I did find a reliable workaround.

Root Cause

By default, Jsoup limits the response body size to 1 MB. If the page content exceeds this limit, the response gets truncated—leading to incomplete data.

Solution

To resolve this, I updated the request configuration to remove the body size limit using the maxBodySize method:

So I tweaked the above code and added maxBodySize which takes size as an integer value.

Document doc = Jsoup.connect(performanceTabURL).maxBodySize(0).get();

Setting the size to 0 removes any restriction on the response body size.

Outcome

With this change, I was able to consistently fetch the complete page data across environments without worrying about size limitations.

If you're working with large HTML responses using Jsoup, this is a small but important tweak that can save you from subtle and hard-to-debug issues.