SPQL is implemented in the ‘Xperimental Scandata Search’ raw API.
The API is available via the following URL:
https://api.silentpush.com/api/v1/merge-api/explore/scandata/search/raw
SPQL is not currently supported within the Silent Push UI, in the Query Builder section. It is only possible to use SPQL in a POST request using tools such as curl or Postman.
Calling the API
To call the API, send a POST request with a body containing a JSON object with the following structure:
{“query”: “<query>”, “fields”:[“<field1>”,”<field2>”,…,”<fieldn>”], “sort”: [“<field1>/<order>”,”<field2>/<order>”,…,”<fieldn>/<order>”]}
The query key must contain the SPQL query as a single string in JSON encoding (i.e., double quotes must be escaped).
Additional API rules
limit=<n>, skip=<n>, and with_metadata=1 must be given as URL parameters:
https://api.silentpush.com/api/v1/merge-api/explore/scandata/search/raw?limit=100&skip=0&with_metadata=1
with_metadata=1 returns an additional metadata key (together with the results), containing information about the matching results, including a job_id which can be used to recall the results from the search for up to 30 days.
To recall search results using the job_id, use the job_id API:
https://api.silentpush.com/api/v1/merge-api/explore/job/0005283b-8094-426a-b653-61bfa9c02d6f
Fields key
The fields key is optional.
If given, it must include a list containing one or more fields from the list of supported field names.
The results from the query will be returned containing only the field names specified in the fields key. This is particularly useful for integration and automation purposes, where only specific data elements are of interest.
If no fields key is included, all available fields for each record in the result will be returned.
Sort key
The sort key is optional.
If given, sort keys must include a list containing one or more fields from the below list of supported field names, followed by forward slash (/), and the sort order (asc or desc).
If only a field name is given, without a sort order, the default sort order is asc.
If a field name given in the sort key is not included in the field name given in the fields key, the field name will not be used in the sort order of the returned results.
If more than one field name is given, nested sorting will be applied to the results, with the first criteria in the sort key being the primary sort criteria, followed by the second criteria in the sort key, and so on.
If sort is not given, the default sort order will be applied: [“scan_date/desc”,”domain/asc”]
API call examples
Example 1
Query returning all fields with JSON encoded/escaped query condition.
URL: https://api.silentpush.com/api/v1/merge-api/explore/scandata/search/raw?limit=100
curl -ks -X POST -H ‘content-type:application/json’ -H ‘x-api- key:<api_key>’
-d ‘{"query": "htmltitle = \"DDoS* not configured\" AND response > 200 AND header.server = ddos*"}’
Example 2
Description: Query returning only specified fields.
URL: https://api.silentpush.com/api/v1/merge-api/explore/scandata/search/raw?limit=100
curl -ks -X POST -H ‘content-type:application/json’ -H ‘x-api-key: <api_key>’
-d ‘{"query": "htmltitle = \"DDoS* not configured\" AND response = 200 AND header.server = ddos*", "fields": ["domain", "scan_date", "hostname", "port", "origin_port", "ip", "origin_ip", "datahash", "header.server", "header.content-type", "htmltitle", "response", "html_body_ssdeep", "ssl.sans", "tld", "ssl.not_before", "ssl.not_after", "ssl.SHA1", "url"]}’
Example 3
Description: RE2 encoded as json (double-backslash as a literal identifier).
URL: https://api.silentpush.com/api/v1/merge-api/explore/scandata/search/raw?limit=100
curl -ks -X POST -H ‘content-type:application/json’ -H ‘x-api-key: <api_key>’
-d {"query": "scan_date > now-30d and ssl.sans ~= /^silent[a-z].+\\.io$/ and domain !~= /silent.*/ and domain ~= /.+/ and ip = [13.249.0.0/16, 52.84.0.0/16]"}
Example 4
Description: Explicit, nested sort order (first by domain in ascending order, then by hostname in ascending order, and finally by scan_date in descending order).
URL: https://api.silentpush.com/api/v1/merge-api/explore/scandata/search/raw?limit=100
curl -ks -X POST -H ‘content-type:application/json’ -H ‘x-api-key: <api_key>’
-d ‘{"query": "domain = mandiant.* AND domain != mandiant.com AND scan_date > now-30d", “sort”: [“domain/asc”,”hostname/asc”,”scan_date/desc”]}’
Example 5
Search for domain name not in .com TLD, scanned less than 30 days ago.
URL: https://api.silentpush.com/api/v1/merge-api/explore/scandata/search/raw?limit=100
curl -ks -X POST -H ‘content-type:application/json’ -H ‘x-api-key: <api_key>’
-d ‘{"query": "domain = mandiant.* AND domain != mandiant.com AND scan_date > now-30d"}’
Example 6
Search for content-type string on servers that are not Apache or nginx, in either network 83.143.113.0/16 or 92.205.3.0/16 with an opendirectory and cpanel in the SSL certificate names scanned less than 10 days ago.
URL: https://api.silentpush.com/api/v1/merge-api/explore/scandata/search/raw?limit=100
curl -ks -X POST -H ‘content-type:application/json’ -H ‘x-api-key: <api_key>’
-d ‘{"query": "header.content-type = \"text/html; charset=iso-8859-1\" AND header.server != Apache AND header.server != nginx AND ip = [83.143.113.0/16, 92.205.3.0/16] AND opendirectory = true AND ssl.sans = cpanel* and scan_date > now-10d"}’