A Data Science Central Community
It’s an interesting problem with no clear answers. And there are perimeters, but a lot of the answers will depend on how they’re broken down. In a traditional network security model, we have hardware based firewalls and intrusion prevention system, but cloud systems tend to use software based firewalls and IDS or IPS, mainly because that’s that only realistic option. (Whether a cloud provider wants to or not, they generally have perimeters in order to comply with PCI-DSS requirements.) Even then, that generally provides no protection against intra-machine network attacks, much less lower level attacks. Firewall and IPS/IDS logs are traditionally where a lot of forensic analysis takes place.
Analytics on network traffic logs is really in its infancy. The most widely deployed system is probably Cisco’s Global Correlation service for their IPS sensors. I’ve seen some web application firewall vendors claim that their systems can generate baseline traffic profiles and alert to exceptions to those profiles. The thing is, none of the vendors in this space are really willing to talk about how their systems work. Hard to know if they really work, to what extent they work, etc.
Sad fact: 86% of breaches are discovered by third parties. (Source: Verizon Business 2011 Data Breach Investigations Report.)
One nice thing about the transition to cloud-type computing is that, fundamentally, none of the technologies are new. Economics is driving companies to use existing technologies in new ways. Hopefully that means that existing security technologies can be adapted to them.
Can I ask how the client instances are divided up? You state that you’re concerned with sifting through “requests,” I’m assuming you’re referring to network based requests? What kind of data do you have to work with?
The client instances on this cloud are mostly deep analytics clusters, some Hadoop, some custom MPI MPMD workflows, interspersed with web applications. The 'requests' I was thinking to have to sort through are the web application requests poking around the network to discover services and weaknesses in customer applications. So the key attack mode I worry about is a customer getting compromised, and the attacker using the compromised customer to try to backdoor into another tenant's system. As this can come from any angle, I haven't thought of a way to segment the traffic as the services can come and go and thus the weaknesses and by extension the attack vectors will come and go.
Obviously, it’s hard to make general statements without knowing a lot about how the machines divide client instances, network infrastructure, etc. My guess is that your best hope is that the machines have host based firewalls that log to a central syslog server. As far as the web requests go, same thing - hopefully there’s some sort of web application firewall like modsecurity that is logging to syslog. From there, look for unusual event codes, unusual ports, etc. A lot of what’s out there revolves around generating alerts based on scoring traffic compared to a baseline… Also look at rules based on time correlations - if event a is always followed by event b and all of a sudden event b isn’t happening after event a, that’s an alert. Obviously, flooding and portscanning are events. Your adaptive prevention will likely have to revolve around generating and deploying updated firewall blacklist rules based on these alerts.