This article covers the primary SPQL data sources used for web scanning and related analysis: Web Scan, Torscan (Dark Web), Web Resources, Services (Banner Search), Open Directory, and WHOIS. These sources provide the foundation for content matching, infrastructure fingerprinting, similarity detection, and domain ownership lookups.
Web Scan, Torscan, and Browser Scan
Shared fields include body analysis, geoip, headers, SSL, favicon, adtech, redirects, and HTML metadata.
Field | Type | Operators | Wildcard / Regex | Description |
|---|---|---|---|---|
domain | string | =, !=, ~=, !~= | Yes | Final domain name |
hostname | string | =, !=, ~=, !~= | Yes | Hostname of the scanned host |
ip | ip | =, != | No | IP address |
url | string | =, !=, ~=, !~= | Yes | Final URL after redirects |
htmltitle | string | =, !=, ~=, !~= | Yes | HTML title |
html_body_length | number | =, !=, >, <, >=, <= | No | Length of the HTML body |
body_analysis.body_sha256 | string | =, != | No | SHA-256 of HTML body |
body_analysis.footer_sha256 | string | =, != | No | SHA-256 of footer |
body_analysis.js_sha256 | string | =, != | No | SHA-256 of JavaScript content |
body_analysis.language | string | =, !=, ~=, !~= | Yes | Detected language |
favicon_md5 | string | =, != | No | MD5 of favicon |
favicon_murmur3 | string | =, != | No | Murmur3 hash of favicon |
jarm | string | =, != | No | JARM TLS fingerprint |
hhv | string | =, != | No | Host Hash Value (header keys) |
ssl.expired | boolean | =, != | No | Whether the certificate is expired |
ssl.issuer.common_name | string | =, !=, ~=, !~= | Yes | Certificate issuer common name |
ssl.subject.organization | string | =, !=, ~=, !~= | Yes | Certificate subject organization |
geoip.country_code2 | string | =, !=, ~=, !~= | Yes | ISO country code |
geoip.asn | number | =, !=, >, <, >=, <= | No | Autonomous System Number |
header.server | string | =, !=, ~=, !~= | Yes | Server header value |
scan_date | datetime | =, !=, >, <, >=, <= | No | Date of the scan |
datahash | string | =, != | No | Unique hash of the scan result |
Web Resources
Field | Type | Description |
|---|---|---|
sha256 | string | SHA-256 of the resource content |
datahash | string | Hash linking the resource to its parent scan |
filesize | number | Size of the resource in bytes |
header.content-type | string | Content-Type header of the resource |
resource_hostname | string | Hostname that served the resource |
resource_url | string | Full URL of the resource |
scan_date | datetime | Date the resource was observed |
Services (Banner Search)
Field | Type | Description |
|---|---|---|
banner | string | Raw service banner |
port | number | Port number |
fingerprints.RSA | string | RSA public key fingerprint |
fingerprints.ECDSA | string | ECDSA public key fingerprint |
ssl.expired | boolean | Whether the TLS certificate is expired |
geoip.asn | number | ASN of the host |
Open Directory
Field | Type | Description |
|---|---|---|
dir | boolean | Whether the entry is a directory |
name | string | File or directory name |
size | number | Size in bytes |
last_modified | datetime | Last modified timestamp |
scheme | string | URL scheme |
WHOIS and Full WHOIS
Field | Type | Description |
|---|---|---|
domain | string | Domain name the record is for |
created | datetime | Original registration date |
expires | datetime | Expiration date |
updated | datetime | Last update date |
registrar | string | Registrar name |
organization | string | Organization name |
string (array) | Registrant email (matches any element) | |
nameserver | string (array) | Nameserver (matches any element) |
country | string | Country of the registrant |
address | string | Street address |
city | string | City |
state | string | State / region |
zipcode | string | Postal code |