07/09/2021 | News release | Distributed by Public on 07/09/2021 10:23
Host List Detection is your subscription's list of hosts and their corresponding up-to-date detections including:
After extracting Host List Detection vulnerability data from Qualys, you'll be able to create custom reporting, perform ad-hoc vulnerability analysis or distribute the vulnerability state of your systems to a central data store. To help achieve this, we are bringing together KnowledgeBase API and Host List API to demonstrate how they work together with Host List Detection API. These three Vulnerability Management (VM) APIs are brought together to provide a rich set of vulnerability information, including:
In Part 3 of this series our goal is to combine the data from Host List, KnowledgeBase, and Host List Detection into the latest, timestamped, point-in-time SQLite database. To help customers realize this goal, we are providing a blueprint of example code called QualysETL that is open-sourced for your organization to develop with. In the diagram below, QualysETL is depicted as a workflow from which you can use the resulting SQLite database for analysis on your desktop, or as part of a continuous live data feed to update your corporate data store in the cloud or your local data center. The accompanying video presents QualysETLin more detail, along with live examples to help you effectively Extract, Transform, Load and Distribute Qualys Data.
The Qualys API is a key component in the API-First model. From the beginning of Qualys in 1999, a rich set of Qualys APIs have been available and continue to improve. As a result, customers have been able to automate processing Qualys in new ways, increasing their return on investment (ROI), and improving overall mean time to remediate (MTTR) vulnerabilities throughout the enterprise.
Even with all these advances in API, some customers continue to experience suboptimal performance in various areas such as automation. So, what are the inherent automation challenges to ETL or Extract, Transform and Load your Qualys Data? And what do we mean by ETL?
To help customers with ETL, we are providing a reusable blueprint of live example code called QualysETL. You can reuse and customize QualysETL example code to suit your organization's needs. In the accompanying video presentation, we will demonstrate installation and operation of the QualysETL software within a Python Virtual Environment on an Ubuntu 20.04 VM.
At the end of this Qualys Host List Detection API blog post and video, you will gain experience in the areas of development, design, and performance with the Qualys API including:
In the next part of this series, we'll add CyberSecurity Asset Management API (formerly known as Global IT Asset Inventory) so you can add a deeper asset inventory correlation of your systems with vulnerability data, including software inventory, end of life, cloud provider information, tagging and other metadata you'll use to enhance the overall security view of your systems.
Qualys Host List Detection: Your subscriptions list of hosts and corresponding up-to-date detections including 1) Confirmed Vulnerabilities, 2) Potential Vulnerabilities and 3) Information Gathered about your system.
With any API, there are inherent automation challenges. Some of those automation challenges for Host List Detection are:
You will want to transform XML data into a format suitable for storage or future correlations with other corporate data sources. The QualysETL blueprint of example code can help you with that objective.
The ETL Design Pattern or Extract, Transform and Load design pattern is a wonderful place to start when transforming Qualys API data into a form/format that is appropriate for your organization. In the diagram, you see depicted the generalized ETL cycle for, the KnowledgeBase which includes rich details related to each vulnerability, the Host List, which is the programmatic driver using Host IDs and VM_Processed_After Date to ETL Host List Detection.
These ETLs are encapsulated in the example blueprint code QualysETL. QualysETL is a fantastic way to get started with your extract, transform and load objectives.
The Host List Detection Activity Diagrams key point is to depict the three types of ETLs, operating simultaneously, resulting in an ETL of all three types of data, Host List, KnowledgeBase, and Host List Detection.
In the diagram you see the ETL of Knowledgebase, operating simultaneously next to the ETL of Host List, which is the programmatic driver for, the ETL of Host List Detection. Near the center of the Activity Diagram, you can see the prepare HostID queue. This is the list of HostIDs that drive the downloading of Host List Detection via spawning of concurrently running jobs through a multiprocessing facility. The result will be CSV, JSON and SQLite which includes the relevant KnowledgeBase, Host List and Host List Detection tables. These data are being stored in both their independent data locations as well as combined into one SQLite database instance that can be used as the most recent view of your vulnerability data. You can distribute snapshots of your ETL data for desktop analysis or as a pipeline of continues updates in your corporate data store.
QualysETL is a blueprint of example code written in python that can be used by your organization as a starting point to develop your companies ETL automation. Below you see the QualysETL Workflow which includes:
One example of distribution would be for your organization to develop a method of uploading a timestamped version of SQLite into an AWS (Amazon Web Services) Relational Database Service or distribute to an AWS S3 Bucket. Another example of distribution would be to ensure the SQLite database is available via a local share on your network where analysts can process and report on vulnerabilities in your organization using their desktop tool of choice.
To install QualysETL, we recommend you spin up a secure virtual machine instance of Ubuntu 20.04 that has connectivity to the internet. When that step is completed, you can login to your Ubuntu instance and work along with me in the accompanying videoto install the application and run your first ETL. The instructions are located on Pypi.org at:
https://pypi.org/project/qualysetl/
Once you have worked along with me in the accompanying video, you can run your own SQL queries to analyze the data and tune the application to meet your needs.
Going forward, here are some final key tips:
The Qualys API Best Practices Technical Series is designed for stakeholders or programmers with general knowledge of programming who want to implement best practices to improve development, design, and performance of their programs that use the Qualys API.
The API Best Practices Series will expand over the coming months to cover other key aspects of the Qualys API, with each presentation building on the previous one and in aggregate providing an overall best practice view of the Qualys API. The next presentations in the series will focus on CyberSecurity Asset Management (CSAM) API formerly known as Global IT Asset Inventory API.