Principles¶
IVRE is a network cartography (or network recon) framework.
Purposes¶
IVRE has five purposes (we use this word to refer to the different types of data IVRE handles), which can be stored by one or more backend databases:
data
: associates IP ranges to Autonomous Systems (AS numbers and names), and geographical information (country, region, city), based on data from Maxmind GeoIP. It can be queried using:- Python API: the
db.data
object from theivre.db
module. - Command line: the
ivre ipdata
tool. - Web (JSON) API: the
/cgi/ipdata/<address>
URL.
- Python API: the
nmap
(sometimes also referred to asscans
): contains Nmap, Masscan, Dismap, Zgrab2, ZDNS, Nuclei, httpx, tlsx and dnsx scan results, as well asivre auditdom
results. Each record represents one host seen during one network scan. It can be queried using:- Python API: the
db.nmap
object from theivre.db
module. - Command line: the
ivre scancli
tool. - Web (JSON) API: the
/cgi/scans
and/cgi/scans/*
URLs.
- Python API: the
passive
: contains host intelligence captured from the network using a Zeek dedicated module calledpassiverecon
, p0f and airodump-ng logs. Each record represents one piece of information (e.g., the HTTPServer:
header valueApache
has been seen 10 times on port 80 of host 1.2.3.4). It can be queried using:- Python API: the
db.passive
object from theivre.db
module. - Command line: the
ivre ipinfo
andivre iphost
tools. The latter is dedicated to passive DNS queries. - Web (JSON) APIs: the
/cgi/passive
and/cgi/passivedns
URLs. The latter is dedicated to passive DNS and is compatible with the Common Output Format implemented for example in CIRCL’s PyPDNS.
- Python API: the
view
: contains a consolidated view of hosts based on data fromnmap
andpassive
. The structure of the records is similar tonmap
, but each record represents a host, seen during one or more network scans and/or seen from network captures. It can be queried using:- Python API: the
db.view
object from theivre.db
module. - Command line: the
ivre view
tool. - Web (JSON) API: the
/cgi/view
and/cgi/view/*
URLs. - Web UI: the
/
or/index.html
Web page.
- Python API: the
flow
: contains aggregated network flows, as seen by Zeek, Argus or Netflows (using Nfdump). It can be queried using:- Python API: the
db.flow
object from theivre.db
module. - Command line: the
ivre flowcli
tool. - Web (JSON) API: the
/flows
URL. - Web UI: the
/flow.html
Web page.
- Python API: the
The following (non-exhaustive) figure shows how the data gets from your favorite open-source tools to IVRE’s databases.
Storing data¶
![digraph {
graph [rankdir=LR];
"maxmind.com";
"Nmap";
"Masscan";
"ivre auditdom";
"Zgrab2";
"Zdns";
"Nuclei";
"httpx";
"tlsx";
"dnsx";
"Dismap";
"airodump-ng";
"p0f";
"Zeek";
"Zeek";
"Argus";
"Nfdump";
XML [label="XML scan result"];
JSON [label="JSON scan result"];
CSV_LOG [label="airodump .csv files"];
P0F_LOG [label="p0f output files"];
PASS_LOG [label="passive_recon.log"];
FLOW_LOG [label=".log files"];
FLOWS [label="flow files"];
db_data [label="db.data" shape="box" style="filled"];
db_nmap [label="db.nmap" shape="box" style="filled"];
db_passive [label="db.passive" shape="box" style="filled"];
db_flow [label="db.flow" shape="box" style="filled"];
db_view [label="db.view" shape="box" style="filled"];
"maxmind.com" -> db_data [label="ivre\nipdata"];
"Nmap" -> XML [label="-oX"];
"Masscan" -> XML [label="-oX"];
"ivre auditdom" -> XML;
"ivre auditdom" -> JSON [label="--json"];
"Zgrab2" -> JSON [label="-o"];
"Zdns" -> JSON [label="-o"];
"Nuclei" -> JSON [label="-json -o"];
"httpx" -> JSON [label="-json -o"];
"tlsx" -> JSON [label="-json -o"];
"dnsx" -> JSON [label="-json -o"];
"Dismap" -> JSON [label="-j"];
"airodump-ng" -> CSV_LOG [label="-w"];
"p0f" -> P0F_LOG [label="-o"];
"Zeek" -> PASS_LOG [label="passiverecon"];
"Zeek" -> FLOW_LOG;
"Argus" -> FLOWS;
"Nfdump" -> FLOWS;
XML -> db_nmap [label="ivre\nscan2db"];
JSON -> db_nmap [label="ivre\nscan2db"];
CSV_LOG -> db_passive [label="ivre\nairodump2db"];
P0F_LOG -> db_passive [label="ivre\np0f2db"];
PASS_LOG -> db_passive [label="ivre\npassiverecon2db"];
FLOW_LOG -> db_flow [label="ivre\nzeek2db"];
FLOWS -> db_flow [label="ivre\nflow2db"];
db_passive -> db_view [label="ivre\ndb2view"];
db_nmap -> db_view [label="ivre\ndb2view"];
{
rank = same;
edge[style=invis];
"maxmind.com" -> "Nmap" -> "Masscan" -> "ivre auditdom" -> "Zgrab2" -> "Zdns" -> "Nuclei" -> "httpx" -> "tlsx" -> "dnsx" -> "Dismap" -> "airodump-ng" -> "p0f" -> "Zeek" -> "Zeek" -> "Argus" -> "Nfdump";
rankdir = UD;
}
}](../_images/graphviz-2f6369a26823883990bcb83b9ac3555fffef6cc9.png)
Accessing data¶
The following (also non-exhaustive) figures show how the data gets from IVRE’s databases back into your hands.
![digraph {
db_data [label="db.data" shape="box" style="filled"];
db_flow [label="db.flow" shape="box" style="filled"];
db_nmap [label="db.nmap" shape="box" style="filled"];
web_api_data [label="Web API\n/ipdata"];
web_api_flows [label="Web API\n/flows"];
web_api_scans [label="Web API\n/scans"];
web_ui_flow [label="Web UI\n/flow.html"];
cli_ipdata [label="CLI\nipdata"];
cli_flow [label="CLI\nflowcli"];
cli_scancli [label="CLI\nscancli"];
db_data -> web_api_data;
db_flow -> web_api_flows;
db_flow -> cli_flow;
db_nmap -> web_api_scans;
web_api_flows -> web_ui_flow;
db_data -> cli_ipdata;
db_nmap -> cli_scancli;
}](../_images/graphviz-f3a9d1551609259fb7dbeca90e2770cd757af72a.png)
![digraph {
db_passive [label="db.passive" shape="box" style="filled"];
db_view [label="db.view" shape="box" style="filled"];
web_api_passive [label="Web API\n/passive"];
web_api_passivedns [label="Web API\n/passivedns"];
web_api_view [label="Web API\n/view"];
web_ui_view [label="Web UI /"];
cli_ipinfo [label="CLI\nipinfo"];
cli_iphost [label="CLI\niphost"];
cli_view [label="CLI\nview"];
db_view -> web_api_view;
web_api_view -> web_ui_view;
db_view -> cli_view;
db_passive -> web_api_passive;
db_passive -> web_api_passivedns;
db_passive -> cli_ipinfo;
db_passive -> cli_iphost;
}](../_images/graphviz-895f16ae20541f9b2ee5370c2ec2d9ff25b1a4d9.png)