Discussion:
Suricata 1.4 http keywords in rule options, how does matching occur for http_header?
Vincent Fang
2013-01-14 22:00:15 UTC
Permalink
Since I cannot rely on external ip addresses, I've been testing out a new
rule that examines the http content, specificially the http header. Looking
at the http-keyword page on redmine, this is the new rule I created for
testing

alert http any any -> any any (msg:"rule fired"; content:"businessweek.com";
http_header;)

Looking at the example, it explains it would look at the http header for
any matching and create an alert if it matches. My one question is how
exactly does it do the matching?


For instance, the businessweek.com address can appear in two places based
on what I examined in the wireshark http packets

Host: www.businessweek.com\r\n

or

Referer: http://www.businessweek.com/\r\n

The examples shown in redmine only show it matching to the Host field but
would this same rule also match if the address showed up in the Referer
field? Right now I'm running into issues with Chrome caching which is
fuddling my results so I can't tell based on my testing, but I was
wondering if anyone had any experience and knows what Suricata is doing
with the content matching in the header field.

Also is it possible to specify a wildcard * in the content: "*
businessweek.com" for matching?

The end goal is for me to maybe specify that

content:"Host: *businessweek.com"; http_header;

so that Suricata would match all packets going to this address. I ran a
test with the modified rule

alert http any any -> any any (msg:"rule fired"; content:"*businessweek.com";
http_header;)

and fast.log isn't populating. I'm not sure if I'm doing it wrong or if
wildcards are not supported in Suricata rules.

Vince
Anoop Saldanha
2013-01-15 08:11:11 UTC
Permalink
Post by Vincent Fang
Since I cannot rely on external ip addresses, I've been testing out a new
rule that examines the http content, specificially the http header. Looking
at the http-keyword page on redmine, this is the new rule I created for
testing
alert http any any -> any any (msg:"rule fired"; content:"businessweek.com";
http_header;)
Looking at the example, it explains it would look at the http header for any
matching and create an alert if it matches. My one question is how exactly
does it do the matching?
For instance, the businessweek.com address can appear in two places based on
what I examined in the wireshark http packets
Host: www.businessweek.com\r\n
or
Referer: http://www.businessweek.com/\r\n
The examples shown in redmine only show it matching to the Host field but
would this same rule also match if the address showed up in the Referer
field? Right now I'm running into issues with Chrome caching which is
fuddling my results so I can't tell based on my testing, but I was wondering
if anyone had any experience and knows what Suricata is doing with the
content matching in the header field.
"*businessweek.com" for matching?
No.
Post by Vincent Fang
The end goal is for me to maybe specify that
content:"Host: *businessweek.com"; http_header;
This would work instead, although you are stuck with the subdomain -

content:"Host: www.businessweek.com"; http_header;

Contents can have fixed strings only.

If you'd want to use wildcards you can use pcre -

pcre:"/Host: .*businessweek.com"/H";

The option "H" refers to matching on the http header filed. We have
other such pcre options to match on different http fields/buffers.
Post by Vincent Fang
so that Suricata would match all packets going to this address. I ran a test
with the modified rule
alert http any any -> any any (msg:"rule fired";
content:"*businessweek.com"; http_header;)
and fast.log isn't populating. I'm not sure if I'm doing it wrong or if
wildcards are not supported in Suricata rules.
--
Anoop Saldanha
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Vincent Fang
2013-01-21 20:57:22 UTC
Permalink
Looking at the snort documentation, regex, and what you suggested, I
modified my rule to look like this now and I set the protocol to ip to
catch everything going out of my machine

alert ip $HOME_NET any -> any any (msg:"alert fired"; pcre:"/Host:\s.*?\.
businessweek.com/H";)

and when I start up my suricata with just that 1 rule and visit business
week on the front page, no alerts show. I click on some subsequent links
and the fast.log still doesn't populate. Only after I click on a few more
links do I see the rules fire. But it's significantly less alerts than if I
used the rule

alert ip $HOME_NET any -> any any (msg:"alert fired"; content:"
businessweek.com"; http_header;)

Can anyone explain why I'm not seeing it fire when I visit the front page?

My setup is running wireshark, suricata, and google-chrome on VMPlayer with
the OS being Fedora 17 and the host machine is Windows 7. I can't
understand why the number of alerts isn't matching the number of packets I
see matching in my wireshark with the filter http && ip.src ==
192.168.0.0/16 && http.host contains "businessweek.com"
Post by Vincent Fang
Post by Vincent Fang
Since I cannot rely on external ip addresses, I've been testing out a new
rule that examines the http content, specificially the http header.
Looking
Post by Vincent Fang
at the http-keyword page on redmine, this is the new rule I created for
testing
alert http any any -> any any (msg:"rule fired"; content:"
businessweek.com";
Post by Vincent Fang
http_header;)
Looking at the example, it explains it would look at the http header for
any
Post by Vincent Fang
matching and create an alert if it matches. My one question is how
exactly
Post by Vincent Fang
does it do the matching?
For instance, the businessweek.com address can appear in two places
based on
Post by Vincent Fang
what I examined in the wireshark http packets
Host: www.businessweek.com\r\n
or
Referer: http://www.businessweek.com/\r\n
The examples shown in redmine only show it matching to the Host field but
would this same rule also match if the address showed up in the Referer
field? Right now I'm running into issues with Chrome caching which is
fuddling my results so I can't tell based on my testing, but I was
wondering
Post by Vincent Fang
if anyone had any experience and knows what Suricata is doing with the
content matching in the header field.
"*businessweek.com" for matching?
No.
Post by Vincent Fang
The end goal is for me to maybe specify that
content:"Host: *businessweek.com"; http_header;
This would work instead, although you are stuck with the subdomain -
content:"Host: www.businessweek.com"; http_header;
Contents can have fixed strings only.
If you'd want to use wildcards you can use pcre -
pcre:"/Host: .*businessweek.com"/H";
The option "H" refers to matching on the http header filed. We have
other such pcre options to match on different http fields/buffers.
Post by Vincent Fang
so that Suricata would match all packets going to this address. I ran a
test
Post by Vincent Fang
with the modified rule
alert http any any -> any any (msg:"rule fired";
content:"*businessweek.com"; http_header;)
and fast.log isn't populating. I'm not sure if I'm doing it wrong or if
wildcards are not supported in Suricata rules.
--
Anoop Saldanha
Anoop Saldanha
2013-01-22 10:14:46 UTC
Permalink
Post by Vincent Fang
Looking at the snort documentation, regex, and what you suggested, I
modified my rule to look like this now and I set the protocol to ip to catch
everything going out of my machine
alert ip $HOME_NET any -> any any (msg:"alert fired";
pcre:"/Host:\s.*?\.businessweek.com/H";)
and when I start up my suricata with just that 1 rule and visit business
week on the front page, no alerts show. I click on some subsequent links and
the fast.log still doesn't populate. Only after I click on a few more links
do I see the rules fire. But it's significantly less alerts than if I used
the rule
alert ip $HOME_NET any -> any any (msg:"alert fired";
content:"businessweek.com"; http_header;)
Can anyone explain why I'm not seeing it fire when I visit the front page?
My setup is running wireshark, suricata, and google-chrome on VMPlayer with
the OS being Fedora 17 and the host machine is Windows 7. I can't understand
why the number of alerts isn't matching the number of packets I see matching
in my wireshark with the filter http && ip.src == 192.168.0.0/16 &&
http.host contains "businessweek.com"
Any chance that you accessed the website first with rule containing
"content:"businessweek.com"" loaded, and then restarted the engine
with pcre rule. If so, the connection might have been still alive
from the browser and it would have reused the same old flow and
suricata may not have picked it up(midstream flow).

To rule out any alert discrepancy from the above case, try this - Load
both the rules at once, instead of conducting separate trials for each
rule. So your rules file would now contain -

alert ip $HOME_NET any -> any any (msg:"alert fired";
content:"businessweek.com"; http_header; sid:1;)
alert ip $HOME_NET any -> any any (msg:"alert fired";
pcre:"/Host:\s.*?\.businessweek.com/H"; sid:2;)

Now run it through the traffic. You should see an equal no of alerts
for both sig 1 and sig 2. If you don't, please capture a pcap and
send it over. Will have it checked.
Post by Vincent Fang
Post by Anoop Saldanha
Post by Vincent Fang
Since I cannot rely on external ip addresses, I've been testing out a new
rule that examines the http content, specificially the http header. Looking
at the http-keyword page on redmine, this is the new rule I created for
testing
alert http any any -> any any (msg:"rule fired";
content:"businessweek.com";
http_header;)
Looking at the example, it explains it would look at the http header for any
matching and create an alert if it matches. My one question is how exactly
does it do the matching?
For instance, the businessweek.com address can appear in two places based on
what I examined in the wireshark http packets
Host: www.businessweek.com\r\n
or
Referer: http://www.businessweek.com/\r\n
The examples shown in redmine only show it matching to the Host field but
would this same rule also match if the address showed up in the Referer
field? Right now I'm running into issues with Chrome caching which is
fuddling my results so I can't tell based on my testing, but I was wondering
if anyone had any experience and knows what Suricata is doing with the
content matching in the header field.
"*businessweek.com" for matching?
No.
Post by Vincent Fang
The end goal is for me to maybe specify that
content:"Host: *businessweek.com"; http_header;
This would work instead, although you are stuck with the subdomain -
content:"Host: www.businessweek.com"; http_header;
Contents can have fixed strings only.
If you'd want to use wildcards you can use pcre -
pcre:"/Host: .*businessweek.com"/H";
The option "H" refers to matching on the http header filed. We have
other such pcre options to match on different http fields/buffers.
Post by Vincent Fang
so that Suricata would match all packets going to this address. I ran a test
with the modified rule
alert http any any -> any any (msg:"rule fired";
content:"*businessweek.com"; http_header;)
and fast.log isn't populating. I'm not sure if I'm doing it wrong or if
wildcards are not supported in Suricata rules.
--
Anoop Saldanha
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Vincent Fang
2013-01-23 17:57:20 UTC
Permalink
How should I send the files to you? I did a restart, and then turned on
wireshark and had it capturing on eth0, turned on suricata with both rules
on, and then opened up chrome to visit the businessweek webpage for the
first time and the number of http alerts vs pcre alerts are not the same. I
saved the pcap file along with the logs and then tried to run suricata in
offline mode using the -r option but in this run it generated no data in
the fast.log or http.log.
Post by Vincent Fang
Post by Vincent Fang
Looking at the snort documentation, regex, and what you suggested, I
modified my rule to look like this now and I set the protocol to ip to
catch
Post by Vincent Fang
everything going out of my machine
alert ip $HOME_NET any -> any any (msg:"alert fired";
pcre:"/Host:\s.*?\.businessweek.com/H";)
and when I start up my suricata with just that 1 rule and visit business
week on the front page, no alerts show. I click on some subsequent links
and
Post by Vincent Fang
the fast.log still doesn't populate. Only after I click on a few more
links
Post by Vincent Fang
do I see the rules fire. But it's significantly less alerts than if I
used
Post by Vincent Fang
the rule
alert ip $HOME_NET any -> any any (msg:"alert fired";
content:"businessweek.com"; http_header;)
Can anyone explain why I'm not seeing it fire when I visit the front
page?
Post by Vincent Fang
My setup is running wireshark, suricata, and google-chrome on VMPlayer
with
Post by Vincent Fang
the OS being Fedora 17 and the host machine is Windows 7. I can't
understand
Post by Vincent Fang
why the number of alerts isn't matching the number of packets I see
matching
Post by Vincent Fang
in my wireshark with the filter http && ip.src == 192.168.0.0/16 &&
http.host contains "businessweek.com"
Any chance that you accessed the website first with rule containing
"content:"businessweek.com"" loaded, and then restarted the engine
with pcre rule. If so, the connection might have been still alive
from the browser and it would have reused the same old flow and
suricata may not have picked it up(midstream flow).
To rule out any alert discrepancy from the above case, try this - Load
both the rules at once, instead of conducting separate trials for each
rule. So your rules file would now contain -
alert ip $HOME_NET any -> any any (msg:"alert fired";
content:"businessweek.com"; http_header; sid:1;)
alert ip $HOME_NET any -> any any (msg:"alert fired";
pcre:"/Host:\s.*?\.businessweek.com/H"; sid:2;)
Now run it through the traffic. You should see an equal no of alerts
for both sig 1 and sig 2. If you don't, please capture a pcap and
send it over. Will have it checked.
Post by Vincent Fang
Post by Anoop Saldanha
Post by Vincent Fang
Since I cannot rely on external ip addresses, I've been testing out a new
rule that examines the http content, specificially the http header. Looking
at the http-keyword page on redmine, this is the new rule I created
for
Post by Vincent Fang
Post by Anoop Saldanha
Post by Vincent Fang
testing
alert http any any -> any any (msg:"rule fired";
content:"businessweek.com";
http_header;)
Looking at the example, it explains it would look at the http header
for
Post by Vincent Fang
Post by Anoop Saldanha
Post by Vincent Fang
any
matching and create an alert if it matches. My one question is how exactly
does it do the matching?
For instance, the businessweek.com address can appear in two places based on
what I examined in the wireshark http packets
Host: www.businessweek.com\r\n
or
Referer: http://www.businessweek.com/\r\n
The examples shown in redmine only show it matching to the Host field but
would this same rule also match if the address showed up in the
Referer
Post by Vincent Fang
Post by Anoop Saldanha
Post by Vincent Fang
field? Right now I'm running into issues with Chrome caching which is
fuddling my results so I can't tell based on my testing, but I was wondering
if anyone had any experience and knows what Suricata is doing with the
content matching in the header field.
"*businessweek.com" for matching?
No.
Post by Vincent Fang
The end goal is for me to maybe specify that
content:"Host: *businessweek.com"; http_header;
This would work instead, although you are stuck with the subdomain -
content:"Host: www.businessweek.com"; http_header;
Contents can have fixed strings only.
If you'd want to use wildcards you can use pcre -
pcre:"/Host: .*businessweek.com"/H";
The option "H" refers to matching on the http header filed. We have
other such pcre options to match on different http fields/buffers.
Post by Vincent Fang
so that Suricata would match all packets going to this address. I ran
a
Post by Vincent Fang
Post by Anoop Saldanha
Post by Vincent Fang
test
with the modified rule
alert http any any -> any any (msg:"rule fired";
content:"*businessweek.com"; http_header;)
and fast.log isn't populating. I'm not sure if I'm doing it wrong or
if
Post by Vincent Fang
Post by Anoop Saldanha
Post by Vincent Fang
wildcards are not supported in Suricata rules.
--
Anoop Saldanha
Anoop Saldanha
2013-01-23 18:24:27 UTC
Permalink
* comments inline
Post by Vincent Fang
How should I send the files to you?
You can mail it to me.
Post by Vincent Fang
I did a restart, and then turned on
wireshark and had it capturing on eth0, turned on suricata with both rules
on, and then opened up chrome to visit the businessweek webpage for the
first time and the number of http alerts vs pcre alerts are not the same. I
saved the pcap file along with the logs and then tried to run suricata in
offline mode using the -r option but in this run it generated no data in the
fast.log or http.log.
- have you checked the pcap if you have all the packets from the flow?
- does wireshark show the checksums for the packets as invalid? If so
you can disable checksum-validation(set it to "no") in the conf file
and re-run suricata.
Post by Vincent Fang
Post by Anoop Saldanha
Post by Vincent Fang
Looking at the snort documentation, regex, and what you suggested, I
modified my rule to look like this now and I set the protocol to ip to catch
everything going out of my machine
alert ip $HOME_NET any -> any any (msg:"alert fired";
pcre:"/Host:\s.*?\.businessweek.com/H";)
and when I start up my suricata with just that 1 rule and visit business
week on the front page, no alerts show. I click on some subsequent links and
the fast.log still doesn't populate. Only after I click on a few more links
do I see the rules fire. But it's significantly less alerts than if I used
the rule
alert ip $HOME_NET any -> any any (msg:"alert fired";
content:"businessweek.com"; http_header;)
Can anyone explain why I'm not seeing it fire when I visit the front page?
My setup is running wireshark, suricata, and google-chrome on VMPlayer with
the OS being Fedora 17 and the host machine is Windows 7. I can't understand
why the number of alerts isn't matching the number of packets I see matching
in my wireshark with the filter http && ip.src == 192.168.0.0/16 &&
http.host contains "businessweek.com"
Any chance that you accessed the website first with rule containing
"content:"businessweek.com"" loaded, and then restarted the engine
with pcre rule. If so, the connection might have been still alive
from the browser and it would have reused the same old flow and
suricata may not have picked it up(midstream flow).
To rule out any alert discrepancy from the above case, try this - Load
both the rules at once, instead of conducting separate trials for each
rule. So your rules file would now contain -
alert ip $HOME_NET any -> any any (msg:"alert fired";
content:"businessweek.com"; http_header; sid:1;)
alert ip $HOME_NET any -> any any (msg:"alert fired";
pcre:"/Host:\s.*?\.businessweek.com/H"; sid:2;)
Now run it through the traffic. You should see an equal no of alerts
for both sig 1 and sig 2. If you don't, please capture a pcap and
send it over. Will have it checked.
Post by Vincent Fang
On Tue, Jan 15, 2013 at 3:11 AM, Anoop Saldanha
On Tue, Jan 15, 2013 at 3:30 AM, Vincent Fang
Post by Vincent Fang
Since I cannot rely on external ip addresses, I've been testing out a new
rule that examines the http content, specificially the http header. Looking
at the http-keyword page on redmine, this is the new rule I created for
testing
alert http any any -> any any (msg:"rule fired";
content:"businessweek.com";
http_header;)
Looking at the example, it explains it would look at the http header for
any
matching and create an alert if it matches. My one question is how exactly
does it do the matching?
For instance, the businessweek.com address can appear in two places based on
what I examined in the wireshark http packets
Host: www.businessweek.com\r\n
or
Referer: http://www.businessweek.com/\r\n
The examples shown in redmine only show it matching to the Host field but
would this same rule also match if the address showed up in the Referer
field? Right now I'm running into issues with Chrome caching which is
fuddling my results so I can't tell based on my testing, but I was wondering
if anyone had any experience and knows what Suricata is doing with the
content matching in the header field.
"*businessweek.com" for matching?
No.
Post by Vincent Fang
The end goal is for me to maybe specify that
content:"Host: *businessweek.com"; http_header;
This would work instead, although you are stuck with the subdomain -
content:"Host: www.businessweek.com"; http_header;
Contents can have fixed strings only.
If you'd want to use wildcards you can use pcre -
pcre:"/Host: .*businessweek.com"/H";
The option "H" refers to matching on the http header filed. We have
other such pcre options to match on different http fields/buffers.
Post by Vincent Fang
so that Suricata would match all packets going to this address. I ran a
test
with the modified rule
alert http any any -> any any (msg:"rule fired";
content:"*businessweek.com"; http_header;)
and fast.log isn't populating. I'm not sure if I'm doing it wrong or if
wildcards are not supported in Suricata rules.
--
Anoop Saldanha
--
Anoop Saldanha
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Anoop Saldanha
2013-01-24 02:49:52 UTC
Permalink
There's a mistake in the rules. Can you retry the above experiment
with the rules -

alert ip $HOME_NET any -> any any (msg:"alert fired";
content:".businessweek.com"; http_header; sid:1;)
alert ip $HOME_NET any -> any any (msg:"alert fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:2;)

The first rule content string doesn't have a "Host" in it, so remove
it from the pcre as well. We are basically asking it to give us a
match as long we have a sub-domained businessweek.com in any header.
Does this give you the same amount of alerts?

-----
Next-

Modify sig 1 and sig 2 to include "Host" as well. Your rules file
would now be -
alert ip $HOME_NET any -> any any (msg:"alert fired"; content:"Host:";
content:".businessweek.com"; within:40; http_header; sid:1;)
alert ip $HOME_NET any -> any any (msg:"alert fired";
pcre:"/Host:\s.*?\.businessweek.com/H"; sid:2;)

The content string would try to match on businessweek.com when present
in the host header(more like an approximate). It will try to find the
string ".businessweek.com" within the next 40 bytes after "Host:"
string. Do note that the 40 bytes is an approximate hoping that all
sub-domains's length would be within that range.

Check if you still get the same no of alerts?

-----

Try these above rules loaded, once against your pcap and next against
live traffic and see if you get the same no of alerts.

-----

However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".

On Wed, Jan 23, 2013 at 11:54 PM, Anoop Saldanha
Post by Anoop Saldanha
* comments inline
Post by Vincent Fang
How should I send the files to you?
You can mail it to me.
Post by Vincent Fang
I did a restart, and then turned on
wireshark and had it capturing on eth0, turned on suricata with both rules
on, and then opened up chrome to visit the businessweek webpage for the
first time and the number of http alerts vs pcre alerts are not the same. I
saved the pcap file along with the logs and then tried to run suricata in
offline mode using the -r option but in this run it generated no data in the
fast.log or http.log.
- have you checked the pcap if you have all the packets from the flow?
- does wireshark show the checksums for the packets as invalid? If so
you can disable checksum-validation(set it to "no") in the conf file
and re-run suricata.
Post by Vincent Fang
Post by Anoop Saldanha
Post by Vincent Fang
Looking at the snort documentation, regex, and what you suggested, I
modified my rule to look like this now and I set the protocol to ip to catch
everything going out of my machine
alert ip $HOME_NET any -> any any (msg:"alert fired";
pcre:"/Host:\s.*?\.businessweek.com/H";)
and when I start up my suricata with just that 1 rule and visit business
week on the front page, no alerts show. I click on some subsequent links and
the fast.log still doesn't populate. Only after I click on a few more links
do I see the rules fire. But it's significantly less alerts than if I used
the rule
alert ip $HOME_NET any -> any any (msg:"alert fired";
content:"businessweek.com"; http_header;)
Can anyone explain why I'm not seeing it fire when I visit the front page?
My setup is running wireshark, suricata, and google-chrome on VMPlayer with
the OS being Fedora 17 and the host machine is Windows 7. I can't understand
why the number of alerts isn't matching the number of packets I see matching
in my wireshark with the filter http && ip.src == 192.168.0.0/16 &&
http.host contains "businessweek.com"
Any chance that you accessed the website first with rule containing
"content:"businessweek.com"" loaded, and then restarted the engine
with pcre rule. If so, the connection might have been still alive
from the browser and it would have reused the same old flow and
suricata may not have picked it up(midstream flow).
To rule out any alert discrepancy from the above case, try this - Load
both the rules at once, instead of conducting separate trials for each
rule. So your rules file would now contain -
alert ip $HOME_NET any -> any any (msg:"alert fired";
content:"businessweek.com"; http_header; sid:1;)
alert ip $HOME_NET any -> any any (msg:"alert fired";
pcre:"/Host:\s.*?\.businessweek.com/H"; sid:2;)
Now run it through the traffic. You should see an equal no of alerts
for both sig 1 and sig 2. If you don't, please capture a pcap and
send it over. Will have it checked.
Post by Vincent Fang
On Tue, Jan 15, 2013 at 3:11 AM, Anoop Saldanha
On Tue, Jan 15, 2013 at 3:30 AM, Vincent Fang
Post by Vincent Fang
Since I cannot rely on external ip addresses, I've been testing out a new
rule that examines the http content, specificially the http header. Looking
at the http-keyword page on redmine, this is the new rule I created for
testing
alert http any any -> any any (msg:"rule fired";
content:"businessweek.com";
http_header;)
Looking at the example, it explains it would look at the http header for
any
matching and create an alert if it matches. My one question is how exactly
does it do the matching?
For instance, the businessweek.com address can appear in two places based on
what I examined in the wireshark http packets
Host: www.businessweek.com\r\n
or
Referer: http://www.businessweek.com/\r\n
The examples shown in redmine only show it matching to the Host field but
would this same rule also match if the address showed up in the Referer
field? Right now I'm running into issues with Chrome caching which is
fuddling my results so I can't tell based on my testing, but I was wondering
if anyone had any experience and knows what Suricata is doing with the
content matching in the header field.
"*businessweek.com" for matching?
No.
Post by Vincent Fang
The end goal is for me to maybe specify that
content:"Host: *businessweek.com"; http_header;
This would work instead, although you are stuck with the subdomain -
content:"Host: www.businessweek.com"; http_header;
Contents can have fixed strings only.
If you'd want to use wildcards you can use pcre -
pcre:"/Host: .*businessweek.com"/H";
The option "H" refers to matching on the http header filed. We have
other such pcre options to match on different http fields/buffers.
Post by Vincent Fang
so that Suricata would match all packets going to this address. I ran a
test
with the modified rule
alert http any any -> any any (msg:"rule fired";
content:"*businessweek.com"; http_header;)
and fast.log isn't populating. I'm not sure if I'm doing it wrong or if
wildcards are not supported in Suricata rules.
--
Anoop Saldanha
--
Anoop Saldanha
--
Anoop Saldanha
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Peter Manev
2013-01-24 08:07:22 UTC
Permalink
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
--
Regards,
Peter Manev
Anoop Saldanha
2013-01-24 08:11:16 UTC
Permalink
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Peter Manev
2013-01-24 08:13:10 UTC
Permalink
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
Vincent Fang
2013-01-24 13:32:30 UTC
Permalink
I will run the test again today with your suggestions, sorry for the delay
in responses.
Post by Peter Manev
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
Matt
2013-01-24 14:15:41 UTC
Permalink
I would find that useful, especially if it increases efficiency in the
same way as http_user_agent. Among other things, I use Suricata to
match blacklists of known bad URLs, and all those rules include a
content match for the HTTP Host.

Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a
foolproof way
Post by Anoop Saldanha
Post by Anoop Saldanha
to match on the host header. The right way would be to provide
a new
Post by Anoop Saldanha
Post by Anoop Saldanha
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Anoop Saldanha
2013-01-24 14:44:00 UTC
Permalink
Sound good. Will open a feature request for "http_host" keyword;
I would find that useful, especially if it increases efficiency in the same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Vincent Fang
2013-01-24 16:59:19 UTC
Permalink
Here's the new results, I will run the tests again to see if it's
consistent but using the wireshark filter

http contains "businessweek.com"

there were 75 matches

and in the fast.log there were 138 total alerts from the two new rules you
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines

so they're both the same. Ran suricata in offline mode and the results were
the same so that's good since they're consistent.

Here's a copy of the two rules

alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)

alert ip $HOME_NET any -> any any (msg:"http header rule fired"; content:".
businessweek.com"; http_header; sid:2;)

In the next few runs I also plan to change the protocol to http instead of
ip, and I technically should get the same numbers yes?
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
Post by Matt
I would find that useful, especially if it increases efficiency in the
same
Post by Matt
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Post by Matt
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Anoop Saldanha
2013-01-25 03:17:20 UTC
Permalink
Here's the new results, I will run the tests again to see if it's consistent
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules you
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results were
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead of
ip, and I technically should get the same numbers yes?
Yes, you should.

Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
I would find that useful, especially if it increases efficiency in the same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Anoop Saldanha
2013-01-25 13:19:30 UTC
Permalink
Try this patch out(you can apply the patch using "git am -3 <patch>")

It introduces a new keyword + pcre modifier that would inspect just
the host header.

The keyword being "http_host" and the pcre modifier being "W"

You can now use it in a rule like this -

alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)

Let me know how it works with the above rules.
Post by Anoop Saldanha
Here's the new results, I will run the tests again to see if it's consistent
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules you
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results were
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead of
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
I would find that useful, especially if it increases efficiency in the same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
--
Anoop Saldanha
Brian Rectanus
2013-01-25 14:27:41 UTC
Permalink
Hi Anoop,

I just glanced at the patch and I think it is only looking in the HTTP
header for the "host" header, but this is incorrect. as the host can
be in the request line as well. You should be looking at the
normalized hostname in the htp_tx_t structure:

tx->parsed_uri->hostname

Additional unit tests should be there to test these, where hostname
should be "www.foo.com" in all cases:

# HTTP/1.0 with no host header
GET http://www.foo.com/ HTTP/1.0

# HTTP/1.1 with correct host data in both url and header
GET http://www.foo.com/ HTTP/1.1
Host: www.foo.com

# HTTP/1.1 with mismatched host data (uri wins)
GET http://www.foo.com/ HTTP/1.1
Host: www.bar.com

# HTTP/1.1 without host header (illegal, but may work)
GET http://www.foo.com/ HTTP/1.1

In addition to those above, you can also have a port in there in each
above, but the hostname should still be "www.foo.com". For example,
add a redundant port 80:

GET http://www.foo.com:80/ HTTP/1.1
Host: www.foo.com:80

And don't forget multiple host headers (also illegal, I think, but
probably will work for many servers) - still "www.foo.com":

GET http://www.foo.com:80/ HTTP/1.1
Host: www.bar.com
Host: another.bogus.host:1234

Cheers!
-B
Post by Anoop Saldanha
Try this patch out(you can apply the patch using "git am -3 <patch>")
It introduces a new keyword + pcre modifier that would inspect just
the host header.
The keyword being "http_host" and the pcre modifier being "W"
You can now use it in a rule like this -
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)
Let me know how it works with the above rules.
Post by Anoop Saldanha
Here's the new results, I will run the tests again to see if it's consistent
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules you
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results were
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead of
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
I would find that useful, especially if it increases efficiency in the same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
--
Anoop Saldanha
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
William Metcalf
2013-01-25 14:47:08 UTC
Permalink
If the CRLF is normalized out from a rule writers perspective we probably continue to use http_header in most cases as generally you want to match on say google.com what you really want to match is .google.com\r\n and not .somethinggoogle.com.ru. Would it be possible to add a endswith modifier for this normalized buffer?

Regards,

Will
Post by Anoop Saldanha
Try this patch out(you can apply the patch using "git am -3 <patch>")
It introduces a new keyword + pcre modifier that would inspect just
the host header.
The keyword being "http_host" and the pcre modifier being "W"
You can now use it in a rule like this -
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)
Let me know how it works with the above rules.
Post by Anoop Saldanha
Here's the new results, I will run the tests again to see if it's consistent
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules you
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results were
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead of
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
I would find that useful, especially if it increases efficiency in the same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
--
Anoop Saldanha
<0001-Add-support-for-a-new-keyword-to-inspect-http_host-h.patch>
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Will Metcalf
2013-01-25 15:23:12 UTC
Permalink
While we are talking about having len and endswith would be really
useful for at least http_uri, http_user_agent, and http_host_header.
The first for performing exact matches i.e.

content:"Mozilla"; http_user_agent; http_user_agent_len:7;

to match

User-Agent: Mozilla\r\n

or

content:".exe"; http_uri; endswith;

to match

GET /blah/blat/foo.exe HTTP/1.1\r\n


etc... Want a feature request? :)

Regards,

Will

On Fri, Jan 25, 2013 at 8:47 AM, William Metcalf
Post by William Metcalf
If the CRLF is normalized out from a rule writers perspective we probably continue to use http_header in most cases as generally you want to match on say google.com what you really want to match is .google.com\r\n and not .somethinggoogle.com.ru. Would it be possible to add a endswith modifier for this normalized buffer?
Regards,
Will
Post by Anoop Saldanha
Try this patch out(you can apply the patch using "git am -3 <patch>")
It introduces a new keyword + pcre modifier that would inspect just
the host header.
The keyword being "http_host" and the pcre modifier being "W"
You can now use it in a rule like this -
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)
Let me know how it works with the above rules.
Post by Anoop Saldanha
Here's the new results, I will run the tests again to see if it's consistent
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules you
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results were
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead of
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
I would find that useful, especially if it increases efficiency in the same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
--
Anoop Saldanha
<0001-Add-support-for-a-new-keyword-to-inspect-http_host-h.patch>
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Peter Manev
2013-01-25 15:55:39 UTC
Permalink
Post by Will Metcalf
While we are talking about having len and endswith would be really
useful for at least http_uri, http_user_agent, and http_host_header.
The first for performing exact matches i.e.
content:"Mozilla"; http_user_agent; http_user_agent_len:7;
to match
User-Agent: Mozilla\r\n
or
content:".exe"; http_uri; endswith;
to match
GET /blah/blat/foo.exe HTTP/1.1\r\n
etc... Want a feature request? :)
yes !
rmkml
2013-01-25 16:28:18 UTC
Permalink
Hi Will and Peter,

Nice feature!

and today it's a good day for writing to santa claus:
Please add a new feature for decoding server side html ? (http_decode ?)
S_HTTP->C: ... file_data; content:"notes:/"; nocase; http_decode; distance:0;
automaticaly decoding notes:/ or notes%3A/ or notes%3A%2F or %6E%6F%74%65%73%3A%2F or %256E%256F%2574%2565%2573%253A%252F...

Best Regards
Rmkml
http://twitter.com/rmkml
Post by Will Metcalf
While we are talking about having len and endswith would be really
useful for at least http_uri, http_user_agent, and http_host_header.
The first for performing exact matches i.e.
content:"Mozilla"; http_user_agent; http_user_agent_len:7;
to match
User-Agent: Mozilla\r\n
or
content:".exe"; http_uri; endswith;
to match
GET /blah/blat/foo.exe HTTP/1.1\r\n
etc... Want a feature request? :)
yes !
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Anoop Saldanha
2013-01-26 03:22:55 UTC
Permalink
Post by Will Metcalf
While we are talking about having len and endswith would be really
useful for at least http_uri, http_user_agent, and http_host_header.
The first for performing exact matches i.e.
content:"Mozilla"; http_user_agent; http_user_agent_len:7;
to match
User-Agent: Mozilla\r\n
Sounds good.
Post by Will Metcalf
or
content:".exe"; http_uri; endswith;
to match
GET /blah/blat/foo.exe HTTP/1.1\r\n
Can be done with isdataat? content:"exe"; http_uri; isdataat:!0, relative;
Post by Will Metcalf
etc... Want a feature request? :)
Regards,
Will
On Fri, Jan 25, 2013 at 8:47 AM, William Metcalf
Post by William Metcalf
If the CRLF is normalized out from a rule writers perspective we probably continue to use http_header in most cases as generally you want to match on say google.com what you really want to match is .google.com\r\n and not .somethinggoogle.com.ru. Would it be possible to add a endswith modifier for this normalized buffer?
Yes we can add, but it probably doesn't make sense since we might
probably pick the hostname from the uri section as well?

One can still do it using isdataat actually - content:"google.com";
http_header; isdataat:!0, relative;
Post by Will Metcalf
Post by William Metcalf
Regards,
Will
Post by Anoop Saldanha
Try this patch out(you can apply the patch using "git am -3 <patch>")
It introduces a new keyword + pcre modifier that would inspect just
the host header.
The keyword being "http_host" and the pcre modifier being "W"
You can now use it in a rule like this -
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)
Let me know how it works with the above rules.
Post by Anoop Saldanha
Here's the new results, I will run the tests again to see if it's consistent
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules you
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results were
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead of
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
I would find that useful, especially if it increases efficiency in the same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
--
Anoop Saldanha
<0001-Add-support-for-a-new-keyword-to-inspect-http_host-h.patch>
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Will Metcalf
2013-01-26 03:28:36 UTC
Permalink
endswith sure seems a lot cleaner :)...

Regards,

Will
Post by Anoop Saldanha
Post by Will Metcalf
While we are talking about having len and endswith would be really
useful for at least http_uri, http_user_agent, and http_host_header.
The first for performing exact matches i.e.
content:"Mozilla"; http_user_agent; http_user_agent_len:7;
to match
User-Agent: Mozilla\r\n
Sounds good.
Post by Will Metcalf
or
content:".exe"; http_uri; endswith;
to match
GET /blah/blat/foo.exe HTTP/1.1\r\n
Can be done with isdataat? content:"exe"; http_uri; isdataat:!0, relative;
Post by Will Metcalf
etc... Want a feature request? :)
Regards,
Will
On Fri, Jan 25, 2013 at 8:47 AM, William Metcalf
Post by William Metcalf
If the CRLF is normalized out from a rule writers perspective we probably continue to use http_header in most cases as generally you want to match on say google.com what you really want to match is .google.com\r\n and not .somethinggoogle.com.ru. Would it be possible to add a endswith modifier for this normalized buffer?
Yes we can add, but it probably doesn't make sense since we might
probably pick the hostname from the uri section as well?
One can still do it using isdataat actually - content:"google.com";
http_header; isdataat:!0, relative;
Post by Will Metcalf
Post by William Metcalf
Regards,
Will
Post by Anoop Saldanha
Try this patch out(you can apply the patch using "git am -3 <patch>")
It introduces a new keyword + pcre modifier that would inspect just
the host header.
The keyword being "http_host" and the pcre modifier being "W"
You can now use it in a rule like this -
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)
Let me know how it works with the above rules.
Post by Anoop Saldanha
Here's the new results, I will run the tests again to see if it's consistent
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules you
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results were
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead of
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
I would find that useful, especially if it increases efficiency in the same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
--
Anoop Saldanha
<0001-Add-support-for-a-new-keyword-to-inspect-http_host-h.patch>
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Anoop Saldanha
2013-01-26 15:22:26 UTC
Permalink
Thanks for the review Brian.

Have updated the patch to now use tx->parsed_uri->hostname. Will
cover the unittests for the various hostname cases shortly.

Note: Since the hostname is normalized to lowercase, I'm invalidating
signatures which doesn't have a nocase attached to the http_header
content. Same for pcre.

* This isn't an incremental patch. Have amended to the old commit
instead. You'll have to undo my previous patch, before applying this
one.
Post by Will Metcalf
endswith sure seems a lot cleaner :)...
Regards,
Will
Post by Anoop Saldanha
Post by Will Metcalf
While we are talking about having len and endswith would be really
useful for at least http_uri, http_user_agent, and http_host_header.
The first for performing exact matches i.e.
content:"Mozilla"; http_user_agent; http_user_agent_len:7;
to match
User-Agent: Mozilla\r\n
Sounds good.
Post by Will Metcalf
or
content:".exe"; http_uri; endswith;
to match
GET /blah/blat/foo.exe HTTP/1.1\r\n
Can be done with isdataat? content:"exe"; http_uri; isdataat:!0, relative;
Post by Will Metcalf
etc... Want a feature request? :)
Regards,
Will
On Fri, Jan 25, 2013 at 8:47 AM, William Metcalf
Post by William Metcalf
If the CRLF is normalized out from a rule writers perspective we probably continue to use http_header in most cases as generally you want to match on say google.com what you really want to match is .google.com\r\n and not .somethinggoogle.com.ru. Would it be possible to add a endswith modifier for this normalized buffer?
Yes we can add, but it probably doesn't make sense since we might
probably pick the hostname from the uri section as well?
One can still do it using isdataat actually - content:"google.com";
http_header; isdataat:!0, relative;
Post by Will Metcalf
Post by William Metcalf
Regards,
Will
Post by Anoop Saldanha
Try this patch out(you can apply the patch using "git am -3 <patch>")
It introduces a new keyword + pcre modifier that would inspect just
the host header.
The keyword being "http_host" and the pcre modifier being "W"
You can now use it in a rule like this -
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)
Let me know how it works with the above rules.
Post by Anoop Saldanha
Here's the new results, I will run the tests again to see if it's consistent
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules you
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results were
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead of
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
I would find that useful, especially if it increases efficiency in the same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
--
Anoop Saldanha
<0001-Add-support-for-a-new-keyword-to-inspect-http_host-h.patch>
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
--
Anoop Saldanha
Victor Julien
2013-01-28 10:30:03 UTC
Permalink
Post by Anoop Saldanha
Thanks for the review Brian.
Have updated the patch to now use tx->parsed_uri->hostname. Will
cover the unittests for the various hostname cases shortly.
Note: Since the hostname is normalized to lowercase, I'm invalidating
signatures which doesn't have a nocase attached to the http_header
content. Same for pcre.
I assume you mean http_host content?

Is there a way to get the hostname as it appeared in the request?
Post by Anoop Saldanha
* This isn't an incremental patch. Have amended to the old commit
instead. You'll have to undo my previous patch, before applying this
one.
Post by Will Metcalf
endswith sure seems a lot cleaner :)...
Regards,
Will
Post by Anoop Saldanha
Post by Will Metcalf
While we are talking about having len and endswith would be really
useful for at least http_uri, http_user_agent, and http_host_header.
The first for performing exact matches i.e.
content:"Mozilla"; http_user_agent; http_user_agent_len:7;
to match
User-Agent: Mozilla\r\n
Sounds good.
Post by Will Metcalf
or
content:".exe"; http_uri; endswith;
to match
GET /blah/blat/foo.exe HTTP/1.1\r\n
Can be done with isdataat? content:"exe"; http_uri; isdataat:!0, relative;
Post by Will Metcalf
etc... Want a feature request? :)
Regards,
Will
On Fri, Jan 25, 2013 at 8:47 AM, William Metcalf
Post by William Metcalf
If the CRLF is normalized out from a rule writers perspective we probably continue to use http_header in most cases as generally you want to match on say google.com what you really want to match is .google.com\r\n and not .somethinggoogle.com.ru. Would it be possible to add a endswith modifier for this normalized buffer?
Yes we can add, but it probably doesn't make sense since we might
probably pick the hostname from the uri section as well?
One can still do it using isdataat actually - content:"google.com";
http_header; isdataat:!0, relative;
Post by Will Metcalf
Post by William Metcalf
Regards,
Will
Post by Anoop Saldanha
Try this patch out(you can apply the patch using "git am -3 <patch>")
It introduces a new keyword + pcre modifier that would inspect just
the host header.
The keyword being "http_host" and the pcre modifier being "W"
You can now use it in a rule like this -
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)
Let me know how it works with the above rules.
Post by Anoop Saldanha
Here's the new results, I will run the tests again to see if it's consistent
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules you
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results were
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead of
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
Post by Matt
I would find that useful, especially if it increases efficiency in the
same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a
new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------

_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Vincent Fang
2013-01-28 20:57:03 UTC
Permalink
Wanted to give the results of the latest patch from Anoop on testing.

Wireshark filter:

http && http.host contains "businessweek.com"

wireshark caught 45 packets that matched this criteria

http-test.rules"
alert http $HOME_NET any -> any any (msg:"pcre rule fired"; pcre:"/\.
businessweek.com/iW"; sid:1;)
alert http $HOME_NET any -> any any(msg:"http rule fired"; content:".
businessweek.com"; nocase; http_host; sid:2;)

fast.log had 90 lines
45 from pcre
45 from http

One thing I wanted to comment was Brian saying how for some clients that
still use HTTP/1.0 with no header host field. There is an http keyword in
Suricata http_uri; I'm guessing that would mean that I would need to create
another rule to complement the http_host if I wanted to catch all dns names
of clients going to the businessweek site?

alert http $HOME_NET any -> any any(msg:"http uri rule fired"; content:"
businessweek.com"; http_uri; sid:3;)

This rule seems to work although I have to take off the leading . from the
domain name.
Post by Victor Julien
Post by Anoop Saldanha
Thanks for the review Brian.
Have updated the patch to now use tx->parsed_uri->hostname. Will
cover the unittests for the various hostname cases shortly.
Note: Since the hostname is normalized to lowercase, I'm invalidating
signatures which doesn't have a nocase attached to the http_header
content. Same for pcre.
I assume you mean http_host content?
Is there a way to get the hostname as it appeared in the request?
Post by Anoop Saldanha
* This isn't an incremental patch. Have amended to the old commit
instead. You'll have to undo my previous patch, before applying this
one.
Post by Will Metcalf
endswith sure seems a lot cleaner :)...
Regards,
Will
On Fri, Jan 25, 2013 at 9:22 PM, Anoop Saldanha <
On Fri, Jan 25, 2013 at 8:53 PM, Will Metcalf <
Post by Will Metcalf
While we are talking about having len and endswith would be really
useful for at least http_uri, http_user_agent, and http_host_header.
The first for performing exact matches i.e.
content:"Mozilla"; http_user_agent; http_user_agent_len:7;
to match
User-Agent: Mozilla\r\n
Sounds good.
Post by Will Metcalf
or
content:".exe"; http_uri; endswith;
to match
GET /blah/blat/foo.exe HTTP/1.1\r\n
Can be done with isdataat? content:"exe"; http_uri; isdataat:!0,
relative;
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
etc... Want a feature request? :)
Regards,
Will
On Fri, Jan 25, 2013 at 8:47 AM, William Metcalf
Post by William Metcalf
If the CRLF is normalized out from a rule writers perspective we
probably continue to use http_header in most cases as generally you want to
match on say google.com what you really want to match is .google.com\r\n
and not .somethinggoogle.com.ru. Would it be possible to add a endswith
modifier for this normalized buffer?
Post by Anoop Saldanha
Post by Will Metcalf
Yes we can add, but it probably doesn't make sense since we might
probably pick the hostname from the uri section as well?
One can still do it using isdataat actually - content:"google.com";
http_header; isdataat:!0, relative;
Post by Will Metcalf
Post by William Metcalf
Regards,
Will
Post by Anoop Saldanha
Try this patch out(you can apply the patch using "git am -3
<patch>")
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
It introduces a new keyword + pcre modifier that would inspect just
the host header.
The keyword being "http_host" and the pcre modifier being "W"
You can now use it in a rule like this -
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)
Let me know how it works with the above rules.
On Fri, Jan 25, 2013 at 8:47 AM, Anoop Saldanha <
On Thu, Jan 24, 2013 at 10:29 PM, Vincent Fang <
Post by Vincent Fang
Here's the new results, I will run the tests again to see if it's
consistent
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
Post by Vincent Fang
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new
rules you
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
Post by Vincent Fang
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the
results were
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
Post by Vincent Fang
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http
instead of
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
Post by Vincent Fang
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Vincent Fang
On Thu, Jan 24, 2013 at 9:44 AM, Anoop Saldanha <
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
Post by Matt
I would find that useful, especially if it increases efficiency
in the
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
same
way as http_user_agent. Among other things, I use Suricata to
match
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
blacklists of known bad URLs, and all those rules include a
content
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
On Thu, Jan 24, 2013 at 1:37 PM, Peter Manev <
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a
foolproof way
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
Post by Anoop Saldanha
Post by Anoop Saldanha
to match on the host header. The right way would be to
provide a
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
Post by Anoop Saldanha
Post by Anoop Saldanha
new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for
that?
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
We should probably consult users/rule-writers if such a
keyword would
Post by Anoop Saldanha
Post by Will Metcalf
Post by Will Metcalf
Post by William Metcalf
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Anoop Saldanha
2013-01-31 16:20:16 UTC
Permalink
Post by Victor Julien
Post by Anoop Saldanha
Thanks for the review Brian.
Have updated the patch to now use tx->parsed_uri->hostname. Will
cover the unittests for the various hostname cases shortly.
Note: Since the hostname is normalized to lowercase, I'm invalidating
signatures which doesn't have a nocase attached to the http_header
content. Same for pcre.
I assume you mean http_host content?
Is there a way to get the hostname as it appeared in the request?
Yes, we can. Shall we introduce it as a new keyword? http_raw_host; ?
Post by Victor Julien
Post by Anoop Saldanha
* This isn't an incremental patch. Have amended to the old commit
instead. You'll have to undo my previous patch, before applying this
one.
Post by Will Metcalf
endswith sure seems a lot cleaner :)...
Regards,
Will
Post by Anoop Saldanha
Post by Will Metcalf
While we are talking about having len and endswith would be really
useful for at least http_uri, http_user_agent, and http_host_header.
The first for performing exact matches i.e.
content:"Mozilla"; http_user_agent; http_user_agent_len:7;
to match
User-Agent: Mozilla\r\n
Sounds good.
Post by Will Metcalf
or
content:".exe"; http_uri; endswith;
to match
GET /blah/blat/foo.exe HTTP/1.1\r\n
Can be done with isdataat? content:"exe"; http_uri; isdataat:!0, relative;
Post by Will Metcalf
etc... Want a feature request? :)
Regards,
Will
On Fri, Jan 25, 2013 at 8:47 AM, William Metcalf
Post by William Metcalf
If the CRLF is normalized out from a rule writers perspective we probably continue to use http_header in most cases as generally you want to match on say google.com what you really want to match is .google.com\r\n and not .somethinggoogle.com.ru. Would it be possible to add a endswith modifier for this normalized buffer?
Yes we can add, but it probably doesn't make sense since we might
probably pick the hostname from the uri section as well?
One can still do it using isdataat actually - content:"google.com";
http_header; isdataat:!0, relative;
Post by Will Metcalf
Post by William Metcalf
Regards,
Will
Post by Anoop Saldanha
Try this patch out(you can apply the patch using "git am -3 <patch>")
It introduces a new keyword + pcre modifier that would inspect just
the host header.
The keyword being "http_host" and the pcre modifier being "W"
You can now use it in a rule like this -
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)
Let me know how it works with the above rules.
Post by Anoop Saldanha
Here's the new results, I will run the tests again to see if it's consistent
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules you
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results were
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead of
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
Post by Matt
I would find that useful, especially if it increases efficiency in the
same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content
match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof way
to match on the host header. The right way would be to provide a
new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword would
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------
_______________________________________________
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Victor Julien
2013-02-01 10:39:20 UTC
Permalink
Post by Anoop Saldanha
Post by Victor Julien
Is there a way to get the hostname as it appeared in the request?
Yes, we can. Shall we introduce it as a new keyword? http_raw_host; ?
That would be good ya.
--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------

_______________________________________________
Suricata IDS Users mailing list: oisf-users-***@public.gmane.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
Vincent Fang
2013-01-25 19:53:42 UTC
Permalink
Sorry for the late response again, I had to rerun the tests,without the
patch, for the first case you explained so here is an archive for you and
anyone else who would like to look at the results. I'm mainly trying to
find out why wireshark with the filter I'm using is seeing 40 results while
the http_header and pcre in suricata only sees 36.

Turned off checksum-validation: no in suricata.yaml

Here's a summary again for the first case without the patch Anoop created
http-test.rules:
alert ip $HOME_NET any -> any any (msg:"http header rule fired"; content:"
businessweek.com"; http_header; sid:1;)
alert ip $HOME_NET any -> any any (msg:"pcre rule fired"; pcre:"/.*?
businessweek.com/H"; sid:2;)

wireshark filter:
http contains "businessweek.com" && ip.src == 192.168.32.136

wireshark had 40 matches
http rule had 36 alerts
pcre had 36 alerts

if you examine the packet number in the pcap file from wireshark, these
were the 4 that wireshark thought matched but I couldn't see
businessweek.com in any of the fields in them
663 -> This one had businessweek.com in the URI but not in any of the http
header fields, I'm not sure if this one should count or not
741
971
1043

Excluding those 4, everything seems to make sense. If anyone here is an
expert at wireshark, please take a look at the pcap file and explain what
other filter should I put in place to make it so the number of packets
matches the number of alerts found in suricata or why Wireshark sees more
than suricata, with the fact known that I'm looking at the entire http
header, not just the Host: field.

Now for version 2 where I would use the http header rule

alert ip $HOME_NET any -> any any (msg:"alert fired"; content:"Host:";
content:".businessweek.com"; within:40; http_header; sid:1;)

Suricata is complaining that it is an invalid siganture saying:

http_header seen with a distance or within without a previous http_header
content. Invalidating signature.

I modified the rule to look like this now and both new rules work as follows

alert ip $HOME_NET any -> any any (msg:"header rule fired"; content:"Host:"
http_header; content:"businessweek.com" http_header; within:40; sid:1;)
alert ip $HOME_NET any -> any any (msg:"pcre rule fired"; pcre:"/Host:\s.*?
businessweek.com/H"; sid:2;)

wireshark filter:
http.host contains "businessweek.com" && ip.src == 192.168.32.136

wireshark had 10 matches

fast.log was 20 lines long with
10 for http header
10 for pcre

So the second versions of rules work exactly the way I expected and this
definitely what I'm looking for. I will try out the patched version later
today and give you the results.
Post by Anoop Saldanha
Try this patch out(you can apply the patch using "git am -3 <patch>")
It introduces a new keyword + pcre modifier that would inspect just
the host header.
The keyword being "http_host" and the pcre modifier being "W"
You can now use it in a rule like this -
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)
Let me know how it works with the above rules.
Post by Anoop Saldanha
Post by Vincent Fang
Here's the new results, I will run the tests again to see if it's
consistent
Post by Anoop Saldanha
Post by Vincent Fang
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules
you
Post by Anoop Saldanha
Post by Vincent Fang
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results
were
Post by Anoop Saldanha
Post by Vincent Fang
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead
of
Post by Anoop Saldanha
Post by Vincent Fang
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Vincent Fang
On Thu, Jan 24, 2013 at 9:44 AM, Anoop Saldanha <
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
Post by Matt
I would find that useful, especially if it increases efficiency in
the
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof
way
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword
would
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
Post by Anoop Saldanha
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
--
Anoop Saldanha
Vincent Fang
2013-01-26 01:16:14 UTC
Permalink
I applied the patch in the source code directory patch <
0001-Add-support-for-a-new-keyword-to-inspect-http_host-h.patch

and ran the tests with the new alert rules and the following wireshark
filter:

http.host contains "businessweek.com" && ip.src == 192.168.32.136

and after visiting the website, I had 28 results in wireshark

fast.log had 56 lines
28 were from the http_host rule
28 were from the pcre rule

so far everything looks good. From Brian's response, if it's true that
hostname can appear elsewhere and not in the http header, that would be a
problem but unfortunately I do not know enough on that field and will be
reading up on it. All I can say for now is the patch appears to work as
intended.
Post by Anoop Saldanha
Try this patch out(you can apply the patch using "git am -3 <patch>")
It introduces a new keyword + pcre modifier that would inspect just
the host header.
The keyword being "http_host" and the pcre modifier being "W"
You can now use it in a rule like this -
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\.businessweek.com/W"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_host; sid:2;)
Let me know how it works with the above rules.
Post by Anoop Saldanha
Post by Vincent Fang
Here's the new results, I will run the tests again to see if it's
consistent
Post by Anoop Saldanha
Post by Vincent Fang
but using the wireshark filter
http contains "businessweek.com"
there were 75 matches
and in the fast.log there were 138 total alerts from the two new rules
you
Post by Anoop Saldanha
Post by Vincent Fang
specified
grep -c "http header" fast.log -> 69 lines
grep -c "pcre version" fast.log -> 69 lines
so they're both the same. Ran suricata in offline mode and the results
were
Post by Anoop Saldanha
Post by Vincent Fang
the same so that's good since they're consistent.
Here's a copy of the two rules
alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
alert ip $HOME_NET any -> any any (msg:"http header rule fired";
content:".businessweek.com"; http_header; sid:2;)
In the next few runs I also plan to change the protocol to http instead
of
Post by Anoop Saldanha
Post by Vincent Fang
ip, and I technically should get the same numbers yes?
Yes, you should.
Keep in mind that the above rules can also match on other headers
containing businessweek.com, for example the referer header.
Post by Vincent Fang
On Thu, Jan 24, 2013 at 9:44 AM, Anoop Saldanha <
Post by Anoop Saldanha
Sound good. Will open a feature request for "http_host" keyword;
Post by Matt
I would find that useful, especially if it increases efficiency in
the
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
same
way as http_user_agent. Among other things, I use Suricata to match
blacklists of known bad URLs, and all those rules include a content match
for the HTTP Host.
Matt
On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
However, any of the techniques mentioned above isn't a foolproof
way
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
Post by Anoop Saldanha
Post by Anoop Saldanha
Post by Anoop Saldanha
to match on the host header. The right way would be to provide a new
keyword called "http_host".
Anoop or Vincent would you please put in feature request for that?
We should probably consult users/rule-writers if such a keyword
would
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
Post by Matt
Post by Anoop Saldanha
be useful to them?
--
Anoop Saldanha
sure
--
Regards,
Peter Manev
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
_______________________________________________
http://suricata-ids.org/support/
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Post by Anoop Saldanha
Post by Vincent Fang
Post by Anoop Saldanha
OISF: http://www.openinfosecfoundation.org/
--
Anoop Saldanha
--
Anoop Saldanha
Loading...