JavaScript not executed on certain pages #1048
Description
Summary: JavaScript is not executed on some pages like https://www.boerse-frankfurt.de/aktie/tesla-inc?mic=XFRA. (live only during 8-22 CET)
In Detail: My code reads data from web pages that update themselves continously using JavaScript. Works on most websites, but fails on the above (and Stockopedia for that matter).
When tested with a real browser or my own code base using Selenium/ChromeDriver, the website above initially loads filled with zero values. Its JavaScript then updates the values and puts two entries into local storage. With HtmlUnit, site is stuck with initial zero values, no local storage set.
Request is plain vanilla:
WebClient webClient = new WebClient(); webClient.setAjaxController(new NicelyResynchronizingAjaxController()); webClient.setCssErrorHandler(new SilentCssErrorHandler()); webClient.setJavaScriptErrorListener(new SilentJavaScriptErrorListener()); webClient.setIncorrectnessListener(new SilentIncorrectnessListener()); webClient.getOptions().setPrintContentOnFailingStatusCode(false); webClient.getOptions().setPopupBlockerEnabled(false); webClient.getOptions().setDoNotTrackEnabled(true); webClient.getOptions().setMaxInMemory(2 * 1024 * 1024); webClient.getOptions().setThrowExceptionOnScriptError(false); webClient.getOptions().setThrowExceptionOnFailingStatusCode(true); webClient.getOptions().setHistoryPageCacheLimit(0); webClient.getOptions().setHistorySizeLimit(0); HtmlPage page = (HtmlPage) webClient.getPage(url);