Manufacturing companies leverage business intelligence (BI) to sift through and analyze manufacturing and supply chain data in order to become more efficient and productive organizations. Often, the real hurdle with analytics is ensuring reliable access to relevant data sets. This article describes how to prepare data to yield strategic and operational insights through a combination of data tiering, federation, querying, and visualization.
Consider the scenario of a car manufacturer looking to implement a predictive maintenance program to reduce maintenance costs for its car assembly machines. Establishing an optimal data storage infrastructure is critical to allow them to find correlations between live IoT sensor data and historical maintenance records, thereby gaining insights into maintenance trends and correlating sensor data.
As shown in Figure 1, such a challenge falls under step 3 of our IIoT end-to-end data integration framework: Compute.
Figure 1: Step 3 in end-to-end data integration framework for IIoT.
Read the first, second, and third articles in this series on end-to-end data integration in the context of IIoT.
Figure 2: Architecture overview of data visualization and analytics enabled by MongoDB.
The proposed methodology leverages the different data tiering capabilities of MongoDB covering the full data lifecycle to create a single API access for BI/analytics. Figure 2 summarizes the different MongoDB features and third-party integrations available to take advantage of the volumes of data generated over time for data-driven business insights.
The challenge of data tiering
The car manufacturer in our example would most likely need to differentiate between the different types of data needed for its predictive maintenance model. Here we make a distinction between operational and analytical workloads.
-
Operational workload: Refers to latency-sensitive data that affects functioning of equipment or powers critical applications/processes.
-
Analytical workload: Refers to life and historical data that does not power mission-critical applications but is readily stored and queried for the purpose of reporting, analytics, or training of AI/ML models.
Figure 3 provides a basic illustration of how MongoDB handles workload isolation leveraging MongoDB replica sets to support real-time BI and analytical workloads without additional ETL jobs.
Figure 3: Illustration of workload isolation in MongoDB.
More advanced architecture patterns for workload isolation or data tiering can be achieved through sharding. Although these approaches are suitable for many scenarios, they are still more like hot/warm data because storage and compute are still tightly coupled.
For maximum cost efficiency at the expense of latency, we must consider newer cloud storage options, such as Amazon S3 or other Blob stores, which decouple storage and compute and are perfectly suited to store so-called cold data. The challenge, however, is how to extract the data from hot stores (such as MongoDB), bring it into the cold storage (such as S3) while maintaining the ability to query the data through a single API.
MongoDB provides several options to facilitate fully automated data tiering, including:
-
Online Archive
-
Atlas Data Lake
Online Archive: Rule-based data archiving
Online Archive in MongoDB Atlas provides an automated rule-based mechanism for moving data out of live/hot clusters to more cost-effective/cold storage (for example, Amazon S3 buckets). This feature removes the burden of building and maintaining potentially complex ETL jobs and data purging functionality while allowing users to configure data offloading within a few simple configuration steps.
Online Archive moves data based on criteria specified in archival rules (as shown in Figure 4). In our example of an auto manufacturing company, sensor data is an excellent use case for this type of data tiering. Sensor data is "hot" when it's created and cools down over time with less need for real-time queries. Our car manufacturer can easily configure an archival rule dependent on the timestamp and in combination with the number of days they want to keep the data in the MongoDB cluster.
Figure 4: Animation showing how Online Archive works.
A broad set of MongoDB Atlas customers across industries already uses Online Archive to save storage costs while maintaining query ability across hot and cold data.
With Online Archive, we were able to save an astounding 60% in data storage costs and 70% in cloud backup costs - reducing our overall database spend by 35%.
Martin Löper, Cloud Solutions Architect,
Nesto Software
Although offloading data already provides major cost savings, there is also potential for more efficient data processing on the consumer side by optimizing the data structures and file formats toward more column-oriented analytical queries. For this purpose, MongoDB has recently released a Data Lake feature set (currently in Preview) that allows users to take advantage of new features such as columnar indexing and an optimized analytical file format.
Data Lake: Columnar indexing of database snapshots
Data Lake is MongoDB's offering of a fully managed analytical storage solution that provides the economics of cloud object storage and is optimized for high-performing analytical queries. It works by reformatting data from a backup snapshot of the Atlas cluster and creating partitioned indexes (illustrated in Figure 5).
Figure 5: Diagram showing how Data Lake works.
Fully integrated as part of MongoDB Atlas, Atlas Data Lake is provisioned alongside Atlas clusters with no infrastructure to set up or manage, and no storage capacity to predict, making the user experience, administration, and support easy. Returning to our example of predictive maintenance model development, performing columnar indexing on the collected data will result in high gains for analytical query performance.
Data Federation: Data virtualization made simple
Rarely do business analysts have all the required data in the same place. Often, it's distributed among different domains and data stores as well as in different formats, like JSON, tabular, CSV, Parquet, Avro, and others. This leads to quite a complex landscape with different API languages, which makes it hard to get easy access to data across all these sources. That's where MongoDB's Atlas Data Federation comes in.
Data Federation allows bridging of these data silos by consolidating all the discussed data sources behind a single API without the need for data duplication (Figure 6). Users can group different data sources to virtual databases and collections and query the data with MQL or SQL across the various sources just like talking to a single DBMS. This approach reduces the effort, time-sink, and complexity of pipelines and ETL tools when working with data in different formats. It also allows users to seamlessly query, transform, and aggregate data from one or more data stores (i.e., Atlas cluster, Atlas Data Lake, Amazon S3 buckets, Online Archive, and HTTP endpoints) to create a single virtual database using the full power of the aggregation pipeline (Figure 7).
Figure 6: Diagram showing how Data Federation works in MongoDB Atlas.
Figure 7: Creating a virtual database in the MongoDB Atlas GUI.
Please refer to the documentation for a more detailed description of the process of creating a Federated Database Instance in MongoDB Atlas.
Data Federation endpoints are not just read-only APIs. Results of querying a federated database instance can be stored back in MongoDB clusters or as files in S3 buckets to power other real-time enterprise or end-user applications, or for performing other analytical tasks and visualizations. In the case of our car manufacturer, real-time sensor data and maintenance history can be queried together and made available to an analytical engine training ML models for remaining useful life prediction.
The fastest way to start building compelling visualizations and gaining insight into the data across MongoDB clusters and file-based data sources through federated instances is through the use of Charts, which comes fully integrated in the Atlas product suite.
Data visualization with Charts
Charts provides a quick, simple, and yet powerful way to visualize data with multiple widgets, dynamic filters, and automatic data refresh like you know it from traditional BI tools. Atlas users can connect dashboards created in Atlas Charts with federated databases and perform correlation analytics in a no-code environment.
Charts is fully integrated with the MongoDB Atlas product suite, which means that data sources in Atlas are immediately accessible from the interface, allowing users to add federated databases as a source for a variety of dashboard visualizations. From displaying device sensor data to calculated values for more sophisticated insights, Charts provides widgets and custom fields calculations to achieve effective and insightful visualizations.
Figures 8 and 9 show two examples of dashboards created in Charts showing time series sensor data from a smart factory and Overall Equipment Effectiveness (OEE) along with other manufacturing performance metrics information. Through the use of these powerful visualizations, the car manufacturer can understand the effect of optimal maintenance strategies on overall factory performance.
Figure 8: Sample shop floor monitoring dashboard created in Atlas Charts.
Figure 9: Sample OEE dashboard in Atlas Charts
To harness existing knowledge and skills around familiar and popular BI tools such as Power BI and Tableau, MongoDB has developed Atlas SQL API, which gives users the option to connect SQL-based business intelligence and analytics tools to Atlas through a variety of drivers and connectors including:
-
Tableau Connector
-
Power BI Connector
-
JDBC Driver
-
ODBC Driver
These Atlas SQL connectors and drivers leverage Data Federation functionality, thereby enabling users to query data across Atlas clusters and cloud storage (such as S3 buckets) and to maintain the comfort of existing SQL-based BI tools that they are familiar with.
Getting started is easy using the Atlas SQL API at no cost with the detailed tutorial and the documentation. Register for a free Atlas user account to try it out.
Watch our recorded webinar to see a live demonstration of how Atlas Federated Instances are created and used as a data source for MongoDB Charts and Tableau.
The banking industry has historically relied on legacy, on-premises systems to store critical financial data in a secure and resilient technology fabric. Today, as transactions happen in real-time and customer demands soar, legacy systems fail to support accelerated modernization and restrict innovation. This change has driven banking enterprises to consider transitioning to newer technologies that can plug better capabilities into business-critical systems while preparing them for tomorrow's challenges. By leveraging MongoDB, Gravity9 enables customers to migrate from legacy systems to more sophisticated technologies in a planned and piecemeal technique. In its effort to modernize technologies that are slowly turning obsolete, Gravity9 recognizes and retains the prominence of legacy systems while migrating with minimal disruption to the status quo. This article describes how Gravity9 performed a data offload from a core retail banking platform to MongoDB using a digital decoupling approach, as shown in Figure 1. Figure 1: Technologies used - Kafka Streams, MongoDB Kafka connectors, Apache Avro, Spring WebFlux, Spring Data Reactive MongoDB repository. The challenge Gravity9's client encountered many performance issues with their core banking platform, Temenos 24, backed by Oracle. Their attempt to offload frequently accessed data from the primary system to the Oracle database failed to give the edge of performance and scalability. The platform was being leveraged for analytical and transactional operations and could not cater to both areas as envisioned. In some instances, this approach resulted in customer transaction rejections. Additionally, the client faced the following challenges: The offload platform proved difficult to manage and extend with most of the logic for business processes written in the stored procedures. The data model within the core banking system was not inherently relational, which amplified the complexity of the platform due to the presence of only two columns: RECID for the ID or Unique Key of the record, and XMLRECORD to store the data. The complexities of the old data source compelled the client to adopt a migration strategy and gradually move into MongoDB, which offered better scalability and a flexible document data model while also migrating the corresponding business logic code. The client expected an improvement in overall performance, maintainability, extensibility, and the potential to deliver better customer experiences. The approach Gravity9 leveraged a digital decoupling approach to create message-driven microservices on top of the client's data. The approach was oriented to a continuous migration process executed in parts rather than in one fell swoop. Using this methodology, Gravity9 could help the client move from the old offload system to MongoDB by building fully scalable microservices for each business domain. The biggest advantage to this approach was that the client's legacy systems could continue as usual behind the scenes. Using a CDC pattern on the underlying Oracle database, the data from the core banking system was gathered, and snapshots modified in the core legacy system were published in real time. An event was generated every time a modification was made in the legacy/core system to keep track of changed data. Gravity9 built an application to learn about the new changes, transform the raw XML format into a structured message and refine it as necessary while broadcasting it as a message to other customers, who could process it or store it on the new MongoDB datastore. Specific dictionaries were developed for this purpose with clear directions about field markers and object structure for the transformed objects. The outcome Although the execution was intended to offload only a part of the data stored on the client's old system, the approach used helped build capabilities to support future migrations of larger volumes. The use of MongoDB Atlas for the implementation delivered efficiency and security in data offloading and met the desired level of performance. Consequently, it resolved scalability issues that would otherwise have occurred with on-premises systems. Although the migrated use cases work on MongoDB-based offload, the implementation left the remaining use cases untouched, allowing them to work on the older systems without handicapping the functioning. Seeking to modernize your banking technology environment? Speak to us today to learn how Gravity9 and MongoDB can help you make the best of new technologies and secure your performance-critical financial data.
While the market for electric vehicles (EVs) continues to expand, several barriers to adoption continue to prevent buyers from making the switch. One of the top concerns among potential buyers is access to charging stations. Currently, there are more than 150,000 charging stations in the U.S. and Canada, but you wouldn't know it by driving along a highway or through a densely populated area. That's because unlike gas stations, charging stations are not advertised on the side of highways or with huge commercial signs. Quebec-based startup, ChargeHub , is out to solve this problem. "ChargeHub's mission is to simplify the electric vehicle charging experience," says ChargeHub Co-founder and CTO, Olivier Proulx. "If we simplify it enough, it will increase electric vehicle adoption." With the ChargeHub app, EV owners can locate charging stations anywhere in North America and know if they're available for charging. ChargeHub is also a member of the MongoDB for Startups program, which helps startups build faster and scale further with free MongoDB Atlas credits, one-on-one technical advice, co-marketing opportunities, and access to a vast partner network. Company origins Even though the app is the company's main focus today, it's not how ChargeHub started. Proulx says he and the co-founders worked as consultants in the EV space before EV cars were even on the road. "We were building electric off road vehicles,'' Proulx says. "Clients were asking us, where are the charging stations in Canada? And we thought the easiest thing to do was to build an app that would show that." After putting the app on the app store for free, the number of downloads convinced them that knowing the location of charging stations was a real problem. So, they diverted their attention away from consulting and started putting more effort into the ChargeHub app. Evolving the app It's not just finding a charging station that EV owners find problematic. When you get to a charging station, you need to pay for charging. The market is highly fragmented, Proulx says. In North America, there are currently over 35 operators of charging stations. When you go to use a new one, you have to sign up for an account with the charging operator and deposit funds into the account to pay for charging. With so many different operators, you wind up with multiple accounts, each with a balance. "With ChargeHub," Proulx says, "you can create one account and charge at over 70% of the charging stations in North America." Today, when EV owners find a station using the ChargeHub app, they can find out if there's a port available before they get there. And, once they start a charging session, the app shows that the charging session has started. That's a lot of transactions that have to happen in real time to ensure a seamless user experience. Proulx says MongoDB Atlas , when he compared it against other databases, gave them the performance they needed at a cost that made the decision easy. Building with MongoDB ChargeHub Co-founder and CTO, Olivier Proulx, describes the EV charging experience for attendees at MongoDB World 2022. Proulx says that the choice to build with MongoDB Atlas from the beginning was critical to its early success. "MongoDB Atlas helped us get the product up and running on a stable, scalable platform from day one," Proulx says. "We didn't have to worry about having to migrate later. And it helped us prove our concept without having to spend too much." Getting free Atlas credits from the MongoDB for Startups program also helped. "When you're building a product and going to market, you're trying to save every penny that you can and extend your runway," Proulx says. The security of Atlas was another key consideration. "Having industry-standard security was critical because we work with electric utilities that are very strict on security," Proulx says. "With MongoDB Atlas, being able to check that box from day one was really critical." Like a lot of startups, the ChargeHub team had to be strategic about where it focused its resources. Managing a database was not part of that strategy. "We were a small team, we didn't want to have to run our own hardware, we wanted everything in the cloud as a service," Proulx says. "Being able to focus on building our solution instead of running things was critical for us. And being able to pick our cloud provider was helpful in managing costs." Cloud flexibility was a big factor for the ChargeHub team according to Proulx: "MongoDB makes it really seamless to pick your cloud provider. And they work with all the main cloud providers. It makes our security policies easier to maintain." Leveraging the cloud depends on how well you're able to integrate it into your existing tech stack. MongoDB scored high marks in that regard. "Our tech stack is based on Node.js and JavaScript. The connection with MongoDB and the document model was so seamless," Proulx says. "Even the Query API fits so well with Node and JavaScript. So for us, it was a no-brainer to go with MongoDB." The road ahead ChargeHub's goal is to reach 100% coverage of charging stations in North America. As EV infrastructure expands, and as more people know that a charging station is never that far away, Proulx says people will be less reluctant to choose an EV for their next car. If the feedback he gets from his users is any indication, new EV buyers don't have anything to worry about. "By having an app and a consumer product, you get feedback from your users," Proulx says. "It's so fun to hear from our users who go on road trips and use ChargeHub to go see the mountains and charge on the way. They're so happy they can finally use one app to charge anywhere they want." If you're looking into an electric vehicle or you already have one, download the ChargeHub app for iOS or Android . Or you can try the all new web experience designed specifically for Tesla drivers to use in the Tesla browser. And be sure to reach out to the ChargeHub support channel if you have feedback. They're always looking to improve the app experience. Are you part of a startup and interested in joining the MongoDB for Startups program? Apply now . For more startups content, check out our Built With MongoDB blog collection.
