BlueXP workload factory for GenAI—Insights from visual data | NetApp Blog

As more companies adopt GenAI-powered assistants that are connected to enterprise data, it's important to get insights that are derived from the entirety of the relevant documents. In many cases, answers to user questions may not be found in the text, but are buried in images, graphs, and charts within documents. Early language models could understand only text, but recent advances in foundation models now enable them to extract insights from multimodal data such as images. By incorporating multimodal data, AI-powered applications can provide a more comprehensive understanding of complex scenarios, enhancing the accuracy and depth of insights.

For instance, in healthcare, combining medical images with patient records can lead to more precise diagnoses and personalized treatment plans. And in retail, analyzing product images alongside customer reviews can offer nuanced market trends and consumer preferences.

To illustrate the benefit of handling multimodal data, let's take an example of how to understand energy trends by using the July 2024 Short-Term Energy Outlook (STEO) report from the U.S. Energy Information Administration. This report provides comprehensive and timely information about the current and the projected states of the energy sector. Let's consider an analyst at a utility company who's using a GenAI-powered research assistant application and needs to understand energy pricing and consumption trends. The analyst asks the following questions by using the STEO report as the source dataset.

Question: How did the Henry Hub natural gas spot price change between 2021 and 2024? And what's the forecast for 2025?

If the response is based on only the text that's contained in the report, and it ignores any graphs and images, the analyst gets the following answer.

Answer without visual insights: The Henry Hub natural gas spot price averaged around $5/MMBtu in 2021. It fell to around $2.10/MMBtu in the first half of 2024 (1H24). The forecast is for it to average almost $2.90/MMBtu in the second half of 2024 (2H24) and then rise to an average of $3.30/MMBtu in 2025.

However, if in addition to the text, the images that are included in the STEO PDF are also processed, the answer to the question contains a lot more context and relevant information.

Answer with visual insights: The Henry Hub natural gas spot price started at around $3/MMBtu in early 2021, experienced significant volatility during 2022-2023 with peaks reaching $8-$9/MMBtu, and then declined significantly through 2023, stabilizing around $2-$3/MMBtu in 2024. The forecast for 2025 projects a gradual increase and stabilization of prices at around $3-$4/MMBtu.

As you can appreciate, the second answer provides more relevant insights in responding to the question and describes the trends based on the visual insights that are presented in the document. To get the more detailed answer, the knowledge base extracted insights from the following graph, which is on page 8 of the STEO document.