AppSuite:OX Abuse Shield: Difference between revisions
No edit summary |
No edit summary |
||
Line 115: | Line 115: | ||
end | end | ||
``` | ``` | ||
The main way to track statistics is by using a sliding-time-window based, in-memory stats database, that enables extremely efficient storage and retrieval of three types of statistics: | |||
* Integer - Counting the number of times relevant events happened | |||
* HyperLogLog - Counting the uniqueness or cardinality of a data set | |||
* CountMin - Distinct count of multiple items in a data set | |||
Country-level IPv4 and v6 IP lookups are available, as are a variety of DNS lookup functions, including querying RBLs. | |||
Many more metrics are available to base decisions on, and are documented in the wforce.conf man page, available on any system with Dovecot Anti-Abuse Shield installed. It also ships with wforce.conf.example, which gives many examples of how to configure the service and policy. | |||
= Simple Testing = | |||
To report (if you configured with 'webserver("127.0.0.1:8084", "secret")'): | |||
``` | |||
$ for a in {1..101} | |||
do | |||
curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234'$a'", "success":"false"}' \ | |||
http://127.0.0.1:8084/?command=report -u wforce:secret | |||
done | |||
``` | |||
This reports 101 failed logins for one user, but with different password hashes. | |||
Now to look up if we're still allowed in: | |||
``` | |||
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234"}' \ | |||
http://127.0.0.1:8084/?command=allow -u wforce:super | |||
{"status": -1, "msg": "diffFailedPasswords"} | |||
``` | |||
It appears we are not! | |||
You can also provide additional information for use by Anti-Abuse Shield using the optional "attrs" object. An example: | |||
``` | |||
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", | |||
"pwhash":"1234", "attrs":{"attr1":"val1", "attr2":"val2"}}' \ | |||
http://127.0.0.1:8084/?command=allow -u wforce:super | |||
{"status": 0, "msg": ""} | |||
``` | |||
An example using the optional attrs object using multi-valued attributes: | |||
``` | |||
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", | |||
"pwhash":"1234", "attrs":{"attr1":"val1", "attr2":["val2","val3"]}}' \ | |||
http://127.0.0.1:8084/?command=allow -u wforce:super | |||
{"status": 0, "msg": ""} | |||
``` | |||
There is also a command to reset the stats for a given login and/or IP Address, using the 'reset' command, the logic for which is also implemented in Lua. The default policy for reset is as follows: | |||
``` | |||
function reset(type, login, ip) | |||
sdb = getStringStatsDB("OneHourDB") | |||
if (string.find(type, "ip")) | |||
then | |||
sdb:twReset(ip) | |||
end | |||
if (string.find(type, "login")) | |||
then | |||
sdb:twReset(login) | |||
end | |||
if (string.find(type, "ip") and string.find(type, "login")) | |||
then | |||
iplogin = ip:tostring() .. login | |||
sdb:twReset(iplogin) | |||
end | |||
return true | |||
end | |||
``` | |||
To test it out, try the following to reset the login 'ahu': | |||
``` | |||
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu"}'\ | |||
http://127.0.0.1:8084/?command=reset -u wforce:super | |||
{"status": "ok"} | |||
``` | |||
You can reset IP addresses also: | |||
``` | |||
$ curl -X POST -H "Content-Type: application/json" --data '{"ip":"128.243.21.16"}'\ | |||
http://127.0.0.1:8084/?command=reset -u wforce:super | |||
{"status": "ok"} | |||
``` | |||
Or both in the same command (this helps if you are tracking stats using compound keys | |||
combining both IP address and login): | |||
``` | |||
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "ip":"FE80::0202:B3FF:FE1E:8329"}'\ | |||
http://127.0.0.1:8084/?command=reset -u wforce:super | |||
{"status": "ok"} | |||
``` | |||
You can retrive all the know stats for a given IP or login with the 'getDBStats' command: | |||
``` | |||
$ curl -X POST -H "Content-Type: application/json" --data '{"ip":"127.0.0.1"}'\ | |||
http://127.0.0.1:8084/?command=getDBStats -u wforce:super | |||
{"blacklisted": false, "ip": "127.0.0.1", "stats": {"OneHourDB": {"diffFailedPasswords": 1}}} | |||
``` | |||
There is also a "ping" command, to check the server is up and answering requests: | |||
``` | |||
$ curl -X GET http://127.0.0.1:8084/?command=ping -u wforce:super | |||
{"status": "ok"} | |||
``` | |||
= Console = | |||
Available over TCP/IP, like this: | |||
``` | |||
setKey("Ay9KXgU3g4ygK+qWT0Ut4gH8PPz02gbtPeXWPdjD0HE=") | |||
controlSocket("0.0.0.0:4004") | |||
``` | |||
Launch wforce as a daemon (`wforce --daemon`), to connect, run `wforce -c`. Comes with autocomplete and command history. If you put an actual IP address in place of 0.0.0.0, you can use the same config to listen and connect remotely. | |||
To get some stats, try: | |||
``` | |||
> stats() | |||
40 reports, 8 allow-queries, 40 entries in database | |||
``` | |||
= Load balancing: siblings = | |||
For high-availability or performance reasons it may be desirable to run multiple instances of Anti Abuse Shield. To present a unified view of status however, these instances then need to share the data from reports. To do so, wforce implements a simple knowledge-sharing system. | |||
Tuples received are broadcast (best effort, UDP) to all siblings. The sibling list is parsed such that we don't broadcast messages to ourselves accidentally, and can thus be identical across all servers. | |||
To define siblings, use: | |||
``` | |||
setKey("Ay9KXgU3g4ygK+qWT0Ut4gH8PPz02gbtPeXWPdjD0HE=") | |||
addSibling("192.168.1.79") | |||
addSibling("192.168.1.30") | |||
addSibling("192.168.1.54") | |||
siblingListener("0.0.0.0") | |||
``` | |||
The first line sets the authentication and encryption key for our sibling communications. To make your own key (recommended), run `makeKey()` on the console and paste the output in all your configuration files. | |||
This last line configures that we also listen to our other siblings (which is nice). The default port is 4001, the protocol is UDP. | |||
To view sibling stats: | |||
``` | |||
> siblings() | |||
Address Sucesses Failures Note | |||
192.168.1.79:4001 18 7 | |||
192.168.1.30:4001 25 0 | |||
192.168.1.54:4001 0 0 Self | |||
``` | |||
With this setup, several wforces are all kept in sync, and can be load balanced behind for example haproxy, which incidentally can also offer SSL. |
Revision as of 07:38, 16 June 2016
API status: In Development
Dovecot Anti Abuse Shield (soon to be released)
Dovecot Anti-Abuse Shield is included along with Dovecot Pro and OX App Suite as a component to protect against login/authentication abuse.
Anti-Abuse Shield runs on a cluster of servers, and integrates with both OX App Suite and Dovecot to detect abuse, brute force attacks and also to enforce common authentication/authorization policies across the platform.
Key Features
Features of Dovecot Anti-Abuse Shield include:
- Replicated/clustered architecture – Login reports are shared between all the servers in a cluster so there is a single view of abuse
- Scriptable Policy Language – Using the Lua language, the functionality of the daemon can be extended to record and protect against a large variety of abusive behavior, as well as implement specific customer policies.
- DNS Lookup Feature – For looking up IPs or domains in blacklists
- GepIP Lookup Feature – GeoIP lookups can be made, and incorporated into policy decisions.
- Ratelimiting and Tarpitting – Extremely flexible, these can be enabled and enforced based on IP address, login name, geoip location, time windows, etc.
- Flexible In-Memory Statistics Database – A versatile and extensible in-memory database is used to store statistics information about abuse over time periods from a few minutes to many hours.
- Integration with Customer Authentication/Authorization Systems – Customers can use the open HTTP REST API to benefit from the protection of the anti-abuse daemon in their own authentication/authorization systems.
- Admin Console – To retrieve statistics and query server state
Pricing and availability
Please contact Open-Xchange Sales for further information and pricing details.
In General
The goal of Dovecot Anti-Abuse Shield is to detect brute forcing of passwords across many servers, services and instances, as well as enforce policy for authentication and authorization. In order to support the real world, brute force detection policy can be tailored to deal with "bulk, but legitimate" users of your service, as well as botnet-wide slowscans of passwords.
Here is how it works:
- Report successful logins via JSON http-api
- Report unsuccessful logins via JSON http-api
- Query if a login should be allowed to proceed, should be delayed, or ignored via JSON http-api
Various other API functions are available, please see https://documentation.open-xchange.com/7.8.2/middleware/components/weakforced.html for full API documentation.
OX App Suite and Dovecot's POP/IMAP server are pre-integrated with Dovecot Anti-Abuse Shield. For more information on how to configure them to work with Anti-Abuse Shield, see \<ref for AppSuite doc\> and http://wiki2.dovecot.org/Authentication/Policy.
However it is also aimed to receive message from other services like:
- Other IMAP/POP servers
- Other Webmail logins
- FTP logins
- Authenticated SMTP
- Self-service logins
- Password recovery services
By gathering failed and successful login attempts from as many services as possible, brute forcing attacks can be detected and prevented more effectively.
The service runs as a daemon (called wforce), and can be clustered in a way that report information is shared between all members of the cluster.
Download and Installation
Redhat Enterprise Linux 7 or CentOS 7
If not already done, add the following repositories to your Open-Xchange yum configuration:
[open-xchange-backend] name=Open-Xchange-backend baseurl=https://software.open-xchange.com/products/dovecot_anti_abuse/stable/backend/RHEL7/ gpgkey=https://software.open-xchange.com/oxbuildkey.pub enabled=1 gpgcheck=1 metadata_expire=0m
and run
$ yum update $ yum install weakforce.d
Debian GNU/Linux 8.0
If not already done, add the following repositories to your Open-Xchange apt configuration:
deb https://software.open-xchange.com/products/dovecot_anti_abuse/stable/backend/DebianJessie /
$ apt-get update $ apt-get install weakforce.d
Policies
Configuration and control of policy is almost entirely through a configuration file which is based on the Lua scripting language. There is a sensible default configuration in wforce.conf, and extensive support for crafting your own policies using the Lua scripting language.
Note that although there is a single Lua configuration file, the report and allow functions run in different lua states from the rest of the configuration, thus you cannot share state.
The following sample is from the default wforce.conf file:
``` -- set up the things we want to track field_map = {} -- use hyperloglog to track cardinality of (failed) password attempts field_map["diffFailedPasswords"] = "hll" -- track those things over 6x10 minute windows newStringStatsDB("OneHourDB", 600, 6, field_map) -- this function counts interesting things when "report" is invoked function twreport(lt) sdb = getStringStatsDB("OneHourDB") if (not lt.success) then sdb:twAdd(lt.remote, "diffFailedPasswords", lt.pwhash) addrlogin = lt.remote:tostring() .. lt.login sdb:twAdd(addrlogin, "diffFailedPasswords", lt.pwhash) end end function allow(lt) sdb = getStringStatsDB("OneHourDB") if(sdb:twGet(lt.remote, "diffFailedPasswords") > 50) then return -1, "", "", {} -- BLOCK! end // concatenate the IP address and login string addrlogin = lt.remote:tostring() .. lt.login if(sdb:twGet(addrlogin, "diffFailedPasswords") > 3) then return 3, "tarpitted", "diffFailedPasswords", {} -- must wait for 3 seconds end return 0, "", "", {} -- OK! end ```
The main way to track statistics is by using a sliding-time-window based, in-memory stats database, that enables extremely efficient storage and retrieval of three types of statistics:
- Integer - Counting the number of times relevant events happened
- HyperLogLog - Counting the uniqueness or cardinality of a data set
- CountMin - Distinct count of multiple items in a data set
Country-level IPv4 and v6 IP lookups are available, as are a variety of DNS lookup functions, including querying RBLs.
Many more metrics are available to base decisions on, and are documented in the wforce.conf man page, available on any system with Dovecot Anti-Abuse Shield installed. It also ships with wforce.conf.example, which gives many examples of how to configure the service and policy.
Simple Testing
To report (if you configured with 'webserver("127.0.0.1:8084", "secret")'):
``` $ for a in {1..101} do curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234'$a'", "success":"false"}' \ http://127.0.0.1:8084/?command=report -u wforce:secret done ```
This reports 101 failed logins for one user, but with different password hashes.
Now to look up if we're still allowed in:
``` $ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234"}' \ http://127.0.0.1:8084/?command=allow -u wforce:super {"status": -1, "msg": "diffFailedPasswords"} ```
It appears we are not!
You can also provide additional information for use by Anti-Abuse Shield using the optional "attrs" object. An example:
``` $ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234", "attrs":{"attr1":"val1", "attr2":"val2"}}' \ http://127.0.0.1:8084/?command=allow -u wforce:super {"status": 0, "msg": ""} ```
An example using the optional attrs object using multi-valued attributes:
``` $ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234", "attrs":{"attr1":"val1", "attr2":["val2","val3"]}}' \ http://127.0.0.1:8084/?command=allow -u wforce:super {"status": 0, "msg": ""} ```
There is also a command to reset the stats for a given login and/or IP Address, using the 'reset' command, the logic for which is also implemented in Lua. The default policy for reset is as follows:
``` function reset(type, login, ip) sdb = getStringStatsDB("OneHourDB") if (string.find(type, "ip")) then sdb:twReset(ip) end if (string.find(type, "login")) then sdb:twReset(login) end if (string.find(type, "ip") and string.find(type, "login")) then iplogin = ip:tostring() .. login sdb:twReset(iplogin) end return true end ```
To test it out, try the following to reset the login 'ahu':
``` $ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu"}'\ http://127.0.0.1:8084/?command=reset -u wforce:super {"status": "ok"} ```
You can reset IP addresses also:
``` $ curl -X POST -H "Content-Type: application/json" --data '{"ip":"128.243.21.16"}'\ http://127.0.0.1:8084/?command=reset -u wforce:super {"status": "ok"} ```
Or both in the same command (this helps if you are tracking stats using compound keys combining both IP address and login):
``` $ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "ip":"FE80::0202:B3FF:FE1E:8329"}'\ http://127.0.0.1:8084/?command=reset -u wforce:super {"status": "ok"} ```
You can retrive all the know stats for a given IP or login with the 'getDBStats' command:
``` $ curl -X POST -H "Content-Type: application/json" --data '{"ip":"127.0.0.1"}'\ http://127.0.0.1:8084/?command=getDBStats -u wforce:super {"blacklisted": false, "ip": "127.0.0.1", "stats": {"OneHourDB": {"diffFailedPasswords": 1}}} ```
There is also a "ping" command, to check the server is up and answering requests:
``` $ curl -X GET http://127.0.0.1:8084/?command=ping -u wforce:super {"status": "ok"} ```
Console
Available over TCP/IP, like this:
``` setKey("Ay9KXgU3g4ygK+qWT0Ut4gH8PPz02gbtPeXWPdjD0HE=") controlSocket("0.0.0.0:4004") ```
Launch wforce as a daemon (`wforce --daemon`), to connect, run `wforce -c`. Comes with autocomplete and command history. If you put an actual IP address in place of 0.0.0.0, you can use the same config to listen and connect remotely.
To get some stats, try:
``` > stats() 40 reports, 8 allow-queries, 40 entries in database ```
Load balancing: siblings
For high-availability or performance reasons it may be desirable to run multiple instances of Anti Abuse Shield. To present a unified view of status however, these instances then need to share the data from reports. To do so, wforce implements a simple knowledge-sharing system.
Tuples received are broadcast (best effort, UDP) to all siblings. The sibling list is parsed such that we don't broadcast messages to ourselves accidentally, and can thus be identical across all servers.
To define siblings, use:
``` setKey("Ay9KXgU3g4ygK+qWT0Ut4gH8PPz02gbtPeXWPdjD0HE=") addSibling("192.168.1.79") addSibling("192.168.1.30") addSibling("192.168.1.54") siblingListener("0.0.0.0") ```
The first line sets the authentication and encryption key for our sibling communications. To make your own key (recommended), run `makeKey()` on the console and paste the output in all your configuration files.
This last line configures that we also listen to our other siblings (which is nice). The default port is 4001, the protocol is UDP.
To view sibling stats:
``` > siblings() Address Sucesses Failures Note 192.168.1.79:4001 18 7 192.168.1.30:4001 25 0 192.168.1.54:4001 0 0 Self ```
With this setup, several wforces are all kept in sync, and can be load balanced behind for example haproxy, which incidentally can also offer SSL.