Most of the work of this project has been concluded, a new module needs to be written which can manage the other three modules written: active discovery, passive discovery, and the mapping module.
This manager module is fairly straightforward: initialise whatever files needed, execute the modules whenever needed, and make they continue executing based on the requirements.
This has been written as well, and the project works as the requirements were set.
A sample output is below:
There have been a few challenges involved with making sure Passive and Active Service Discovery modules are writing to a common JSON file (datastore) and are keeping the data current.
There were initial hiccups. Both modules kept overwriting each other’s data after every phase as the writing mode was ‘w’.
Then, later I changed it to append, and that worked fine. But, the problem with this approach was that there would be stale data which may or may not be correct at a later stage of time.
Hence, I choose a different approach entirely. After the initial file was created, every time either of the modules had to write data to it, then they first needed to read the entire datastore JSON file first. Then, verified about the credibility of the data present, check if the data to be written has not already been written. Then, if both the conditions are met, whatever the module writes can be considered current and latest.
Based on the pseudo code written in previous weeks, the mapping module has been successfully completed.
The ideology has been to read data from active and passive service discovery’s modules. For now, a sample JSON file has been written from which the mapping module reads off of.
The map is not dynamic, the current map needs to be closed to generate the latest map.
The nodes are the servers/services in the Application Dependency Map and the edges of this map are relationships between these nodes.
Say, we have a web server, web application and a MySQL server. Then these are the nodes and the edges are:
Web applicatioon -> Web Server
Web application -> MySQL Server
As both Passive and Active service Discovery have been working independently as required, this week I focussed more on how to go about mapping the Application Dependency Map.
I have identified the tools needed to write the mapping module: networkx and matplotlib.
The working of both the modules has been tested, by creating a few basic maps, all manually, which included a bi-directional cyclic map.
As described previously about challenges involved with identifying master and slave MySQL servers, it cannot be said conclusively about which is master and which is slave.
Any repetitive one-way traffic between any two MySQL servers gives us an idea that one of them is the Master server, and the other the slave. But, to verify this, we need to make sure that the traffic is not just between two hosts, but also the ports, i.e. the traffic is beginning on the Master server’s MySQL port and culminating at the slave server’s MySQL port. The port is discovered in the discovery phase and this information is stored then.
- Compare IP of application, and search for same IP in web servers pool. Gives the relationship for ‘Runs on’.
- Check if any traffic between WP host and SQL host. Don’t just see the traffic, but also verify if WP is talking to the port which SQL is on. Gives the relationship for ‘Depends on’.
- The IP of the WP application itself gives the relationship for ‘Hosted on’.
- The relationship ‘Using Port’ is easily found during discovery itself.
- SQL instance, Web server etc can also be easily mapped appropriately.
How to find SQL instances with master-slave architecture?
- Classify the discovered services, by attaching a tag name to each.
- Establish relationships between various services by looking at TCP port, IP address.
- Draw a dependency map after all relationships are established.
Finally, show not only the dependency map, but a detailed list of all established relationships between services, servers, and hosts.
I have extended the module to support a few more applications this week. I had to go back and change the pattern recognition methodology for wordpress and MySQL server (see previous week post) as they were not working for all cases.
Continue reading “Application Service Discovery – Week 8”
I went through the ntopng code that I had mentioned last week, but I couldn’t gain a lot of information from it. So, I decided to dump it and proceed with writing code for my project.
Continue reading “Application Service Discovery – Week 7”
Last week when I was trying to read data from packets, I came across a lot of garbled or unreadable data. I thought I could be unpacking the packets wrong, but I verified that I wasn’t. I realised that that data was just encrypted data and thus, unless I had the session keys as well, I won’t be able to read that data. I was able to verify this after looking at the packets in wireshark, as wireshark can tell us whether the packet was HTTPS or HTTP.
Continue reading “Application Service Discovery – Week 6”