axa
2.2.0
Farsight Security Advanced Exchange Access (AXA)
|
This is the Advanced Exchange Access (AXA) toolkit. It contains tools and a C library to bring Farsight's real-time data and services directly from the Farsight Security Information Exchange (SIE) to the subscriber's network edge.
AXA-based solutions are often preferable over procuring a direct, physical connection to the SIE, usually via a co-located blade.
AXA enables subscribers to connect to Farsight's subscription-based SRA (SIE Remote Access) and RAD (Real-time Anomaly Detector) servers. These servers provide access to data and services built from Farsight's SIE.
SRA streams real-time SIE data while RAD streams real-time anomaly detection data (from services such as Brand Sentry and Domain Sentry).
The axa-tools distribution contains the following:
sratool
: A test/debug/instructional command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data.radtool
: A test/debug/instructional command-line tool used to connect to a RAD server, set watches, enable anomaly detection modules, and stream data.sratunnel
: A production command-line tool that streams SIE data to the local network.radtunnel
: A production command-line tool that streams anomaly data to the local network.libaxa
: A C library providing an API for the AXA protocol including:For usage details on sratool
, radtool
, sratunnel
, and radtunnel
, please see their respective man pages (included in the distribution).
Note that "AXA" is a bit of an overloaded term. Depending on the context, it can refer to actual AXA wire protocol, the C API, the suite of tools presented here, or even a clustered set of SRA and RAD servers. In this document, when apropos, context is provided in order to disambiguate these situations.
Some of the channels offered by the SIE network burst to an extremely high bitrate (some over 500Mbps). AXA has two ways to deal with such network-hungry situations: optional filtering and loss-tolerance built into the protocol.
Filtering can take one of the following forms:
Finally, AXA is a deliberately lossy protocol. If a subscriber requests more data than the network can carry, data overruns will occur. When this happens, loss markers are transmitted reliably within the AXA stream to inform the subscriber via the AXA accounting subsystem. At this point, the subscriber's possible mitigation strategies include:
AXA can built manually or, on Debian systems, installed by using pre-built packages.
The AXA suite has the following external dependencies:
--without-libxs
, be sure to use version >= 0.11.2)Optional dependencies:
After satisfying the above, build with something like:
./autogen.sh
followed by ./configure
and make
To generate the API documentation (including an HTMLized version of this document): make doc
. The HTML documentation will be in the html
directory and can be rendered in any modern browser. Something like open html/index.html
should get you started.
Finally, to give the AXA suite a home, sudo make install
.
The binary packages of AXA and its dependencies are available from a Debian package repository maintained by Farsight Security. These packages should be used instead of building from source on Debian-based systems.
To install the AXA Tools sratool
, radtool
, sratunnel
, radtunnel
:
To install AXA development files (if you wish to use the libaxa C API):
Once axa-tools is installed, you'll need to work with your account manager to decide which transport (described below) is best for your use case and then have your account provisioned.
AXA offers three encrypted transports for setting up sessions and tunneling data:
ECDHE-RSA-AES256-GCM-SHA384
suite offering "currently infeasible to break" encryption and perfect forward secrecy.ECDHE-RSA-AES256-GCM-SHA384
suite offering "currently infeasible to break" encryption and perfect forward secrecy.In order to use AXA, one of these is required. While all three offer commensurate security, Farsight strongly recommends using the apikey transport due to its ease of setup and use.
AXA compresses all NMSGs before transmission across the network using NMSG's built-in compression capability (currently zlib). IP packets are not compressed.
Your Farsight Security account manager will provide you with an alphanumeric apikey string. This apikey is used to both identify you and authenticate your session to the AXA servers.
The AXA apikey transport listens at the following URIs:
apikey:<your_apikey_here>@axa.sie-remote.net,1011
apikey:<your_apikey_here>@axa.sie-remote.net,1012
SRA listens on TCP/1011
and RAD listens on TCP/1012
and both transit standard TLS data.
You can connect as per the following:
Connecting via sratool
Connecting via radtool
Connecting via sratunnel
Connecting via radtunnel
The AXA TLS transport listens at the following URIs:
tls:user_name@sra.sie-remote.net,1021
tls:user_name@rad.sie-remote.net,1022
SRA listens on TCP/1021 and RAD listens on TCP/1022 and both transit standard TLS data.
To setup TLS access for SRA and/or RAD, you need to do the following:
axa_make_cert
: Generate AXA certificate and private key filesaxa_server_cert
: Retrieve the AXA server certificate fingerprintaxa_link_certs
: Create AXA certificate links~~~ # axa_make_cert -u username Create /usr/local/etc/axa/certs? y Generating a 2048 bit RSA private key ............+++ .............+++ writing new private key to 'username.key' ~~~
~~~ # chown user. /usr/local/etc/axa/certs/username.key ~~~
~~~ # axa_server_cert -s axa.sie-remote.net,1021 Obtained certificate for "farsight" with SHA1 Fingerprint=2D:0C:92:23:B9:6F:70:E7:F3:E3:7A:2B:D6:F5:D4:CA:1F:F8:CE:71 Install it in /usr/local/etc/axa/certs/farsight.pem? yes ~~~
username.pem
) to your Farsight Security account manager. DO NOT EVER SHARE YOUR PRIVATE KEY (username.key
). This is the private half of your generate key pair that you should keep safe. As soon as your account is provisioned you will receive notification from Farsight Security.You can connect as per the following:
Connecting via sratool
Connecting via radtool
Connecting via sratunnel
Connecting via radtunnel
The AXA SSH transport listens at the following URIs:
sra-service@sra.sie-remote.net
rad-service@rad.sie-remote.net
Both services listen on TCP/22 for standard SSH traffic.
To setup SSH access for SRA and/or RAD, you need to do the following:
ssh-keygen
: $ ssh-keygen -t rsa -b 4096 -f ~/.ssh/farsight-axa-id_rsa Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/user/.ssh/farsight-axa-id_rsa. Your public key has been saved in /home/user/.ssh/farsight-axa-id_rsa.pub. The key fingerprint is: SHA256...
~/.ssh/config
file to specify the private half of the SSH key pair for the SRA and RAD servers: ~~~ Host sra.sie-remote.net rad.sie-remote.net IdentityFile ~/.ssh/farsight-axa-id_rsa ~~~
~/.ssh/farsight-axa-id_rsa.pub
) to your Farsight Security account manager. DO NOT EVER SHARE YOUR PRIVATE KEY (~/.ssh/farsight-axa-id_rsa
). This is the private half of your generated key pair that you should keep safe. As soon as your account is provisioned you will receive notification from Farsight Security.You can connect as per the following:
Connecting via sratool
Connecting via radtool
Connecting via sratunnel
Connecting via radtunnel
AXA now requires a subscriber-side configuration file used as a convenience to specify session defaults. By default AXA will look for it ~/.axa/config
. Currently it is used to store "connection aliases" that provide a facility to create shortcut mnemonics to specify the AXA server connection string. This is especially useful for long connection strings associated with the apikey transport. It can also be used for the other transports. As it can contain sensitive information, the file must be readable/writable only by "owner" or AXA-based tools will refuse to load.
For example:
After creating the above aliases, you can replace the server connection URI with your shortcut name as per the following:
The AXA config file is shared for sratool
, radtool
, sratunnel
, radtunnel
, and is available via the API. As such, all of the tools (including the ones you may build) have access to these alias shortcuts.
The following are a few examples of how to use sratool
, sratunnel
and radtool
.
Here's a simple example using sratool
to stream five NMSGs seen on the Newly Observed Hostnames (NOH) channel (channel 213):
sra> connect sra-apikey
: we connected to an SRA server using the apikey transport. Under the hood, sratool
consulted the AXA config file to search for the "sra-apikey" alias and extract the server connection URI. The HELLO
response from the remote end provides its version number and the protocol level.sra> count 5
: we asked our sratool
client to stop after emitting five messages.sra> channel 213 on
: we then asked the remote end to listen to SIE channel 213.sra> 1 watch ch=213
: we then asked to watch all content on channel 213 (with no rate limiting or filtering), which is a common choice for 213 since its volume is low (~200kbps).The following example shows how to use sratunnel
to stream NMSGs from NOH to a file and then read the resultant NMSG file with nmsgtool display a single NMSG record as new line delimited JSON using the jq program.
sratunnel -s sra-apikey -c 213 -w ch=213 -o nmsg:file:213.nmsg
: we invoked sratunnel and connected to SRA using the apikey transport. Channel 213 is enabled, and a 213 "all watch" is set. Finally, NMSGs are written to a file. The program runs for some time then we kill it via ctrl-c.$ nmsgtool -c 1 -r 213.nmsg -J - -- | jq .
: The nmsgtool
program is run to read a single NMSG from the output file and pipeline to the jq program to pretty print it.Next, radtool
is used to load the Brand Sentry anomaly module to watch for suspected brand infringement in the Internationalized Domain Names (IDN) namespace for four well-known brands.
rad> connect rad-apikey
: We connected to RAD over the apikey transport using the "rad-apikey" alias. The HELLO
response from the remote end tells us its version number and the protocol level.rad> verbose on
: We turn on verbose mode to get more information about each hit.rad> 1 watch dns=*.
: We set a DNS wildcard "all-watch". This will match all dns hostnames which is what we want for Brand Sentry – we want to look at the entire DNS namespace (except for four level domains, as per below).rad> 1 anomaly brand_sentry brand=facebook,apple,netflix,google matcher=idn_homoglyph whitelist=*.facebook.com,*.apple.com,*.netflix.com,*.google.com
: We switched on the anomaly detector. This command enables the brand_sentry anomaly module looking for Internationalized Domain Names (IDNs) "suspiciously close" to "facebook, google, apple", or "netflix". Hostnames in the *.facebook.com
, *.apple.com
, *.netflix.com
, and *.google.com
are considered safe and are ignored.$ idn -u xn--fcebook-s3a.com.
: We use the GNU libidn idn command line tool to convert the punycode encoded domain into its Unicode representation.With sratunnel
and nmsgtool
, you can create your own local "SIE node". This can be useful if you want cobble together your own local SIE cloud with the channels you're subscribed to. First, invoke sratunnel
as per the following:
Before continuing, we confirm traffic flowing by checking with tcpdump
(on OS X this looks like):
Finally, we use nmsgtool
to receive and decapsulate the data. Here we invoke it to capture and display three payloads:
sratunnel -s sra-apikey -c 204 -w ch=204 -o nmsg:127.0.0.1,8000 &
: invoke sratunnel
and stream all of channel 204 (Passive DNS traffic that has been de-duplicated, filtered, and verified). The output is set to be NMSGs to the loopback interface on port UDP/8000 (UDP is the default NMSG transport protocol).tcpdump -n -c 3 -i lo udp port 8000
: tcpdump is used to verify packets are being received.nmsgtool -l 127.0.0.1/8000 -c 3
: nmsgtool
is invoked to listen on the loopback interface on UDP/8000
and three NMSGs are emitted to stdout.The AXA protocol consists of a pair of streams of messages between a subscriber's client (such as sratool
) and an AXA server (such as SRA), one stream in each direction, currently over a single TCP connection.
AXA must sit on top of a reliable stream protocol such as TCP and so has no provisions to detect or recover from duplicate, out-of-order, lost, or partially lost data. Note that SIE data can be lost before encapsulation into AXA protocol messages due to issues such as network congestion, CPU overload, etc.
As mentioned above, protocols such as TLS or SSH are used below the AXA layer and above TCP to provide authentication, confidentiality, and integrity as shown below.
The authoritative definition of the AXA protocol is contained in the axa/protocol.h file.
Values that originate in SRA or RAD servers such as message lengths use little endian byte order in the AXA protocol. Other values such as IP addresses and port numbers are big endian for consistency with their sources such as host tables. SRA and RAD data such as NMSG messages and IP packets have their original byte orders.
The stream protocols below the AXA protocol are responsible for authentication and authorization. An AXA client and server pair on a computer can use unadorned TCP through the loop-back interface or use a UNIX domain socket. The AXA protocol assumes this is safe.
Between separate computers, the AXA protocol can use UNIX pipes to the stdin
and stdout
streams provided by the ssh
command or the functions of an SSH library such as libssh2
(SSH must identify and authenticate the client and server to each other) or the TLS library.
The AXA client starts by waiting for an AXA_P_OP_HELLO
message from the server. Over a local stream, the client then sends an AXA_P_OP_USER
message to tell the server which parameters to use. When SSH is used, the user name is provided through the SSH protocol.
Every AXA message starts with the following 8 octet header:
len
: A 32-bit value covering the entire length of the AXA message including the header. Many AXA messages are variable length.tag
: A 16-bit identifier used to uniquely "tag" specific events during the lifetime of an AXA session. To refer to these events, the client or server will use this tag. Some AXA messages do not require tags, in that case the tag field should be AXA_TAG_NONE
. Required tags must be unique during the lifetime of the corresponding client request. Some client requests such as a "watch" can last indefinitely and will elicit many server responses all with the same tag.pvers
: An 8-bit protocol version number that allows AXA clients and servers of different ages to find a mutually compatible version of the AXA protocol.op
: The 8-bit op (opcode) specifies an operation requested by the client, a response from the server, or data from the server. The universe of opcodes is discussed below.For a detailed discussions of the AXA protocol message types, see the doxygen generated page for protocol.h
.
The following is an AXA protocol quick reference chart intended for application developers building libaxa programs.
AXA_TAG_NONE
, as described above.OPCODE | VAL | SENT BY | TAG | DESCRIPTION |
---|---|---|---|---|
AXA_P_OP_NOP | 0 | CLIENT / SERVER | NO | carries no data, is intended only to ensure that the TCP connection is still up |
AXA_P_OP_HELLO | 1 | SERVER | NO | helps the client choose a compatible AXA protocol version |
AXA_P_OP_OK | 2 | SERVER | YES | indicates the success of the preceding client request with the same tag |
AXA_P_OP_ERROR | 3 | SERVER | YES | indicates the failure of a preceding client request with the same tag |
AXA_P_OP_MISSED | 4 | SERVER | NO | carries details about data or packet loss due to rate limiting or network congestion |
AXA_P_OP_WHIT | 5 | SERVER (SRA) | YES | reports a "watch hit" or packet or NMSG message that matched an SRA watch with the same tag |
AXA_P_OP_WLIST | 6 | SERVER (SRA) | YES | reports a current watch in response to AXA_P_OP_WGET from the client referenced by tag |
AXA_P_OP_AHIT | 7 | SERVER (RAD) | YES | reports an "anomaly hit" or packet or NMSG message detected by a set of anomaly detector |
AXA_P_OP_ALIST | 8 | SERVER (RAD) | YES | reports a current anomaly detector in response to AXA_P_OP_AGET |
AXA_P_OP_CLIST | 9 | SERVER (SRA) | NO | reports the on/off state and specification of an SRA channel |
AXA_P_OP_USER | 129 | CLIENT | NO | indicates the AXA protocol is used over a local stream and rejected otherwise |
AXA_P_OP_JOIN | 130 | CLIENT | NO | used to bundle TCP connections |
AXA_P_OP_PAUSE | 131 | CLIENT | NO | ask the server to temporarily stop sending packets or NMSG messages |
AXA_P_OP_GO | 132 | CLIENT | NO | ask the server to resume sending packets or NMSG messages |
AXA_P_OP_WATCH | 133 | CLIENT | NO | specify interesting packets or NMSG messages |
AXA_P_OP_WGET | 134 | SERVER (SRA) | – | requests one (with specified tag) or all (tag 0) current watches in AXA_P_OP_WLIST messages |
AXA_P_OP_ANOM | 135 | SERVER (RAD) | YES | specify an anomaly detector |
AXA_P_OP_AGET | 136 | SERVER (RAD) | – | requests one or all current anomaly detectors |
AXA_P_OP_STOP | 137 | CLIENT | NO | ask the server to delete the watch or anomaly detector by tag |
AXA_P_OP_ALL_STOP | 138 | CLIENT | NO | ask the server to delete all watches or anomaly detectors |
AXA_P_OP_CHANNEL | 139 | CLIENT (SRA) | NO | tell the SRA server to enable or disable one channel or all channels |
AXA_P_OP_CGET | 140 | CLIENT (SRA) | NO | get the specifications and states of all channels |
AXA_P_OP_OPT | 141 | CLIENT / SERVER | NO | set various options (rate limiting) report rate limits, how much has been used |
AXA_P_OP_ACCT | 142 | CLIENT / SERVER | NO | request accounting information |
AXA_P_OP_RADU | 143 | SERVER (RAD) | NO | request RAD Unit balance |
See the JSON schema describing the JSON output format.
For a detailed walkthrough of a "Hello World", AXA-style, please see the following Farsight Security Blog articles: