I’ve added a simple donations page for the project where interested parties can help. Please view the page for details, and thanks!
I’ve synced the latest version of the code for DeepMac to Github. In simple terms, I’ve largely abandoned my original codebase (Perl & PHP) in favor of both a new language (Python) and a fundamentally different way of handling the data (see the wiki for more details).
The new codebase is not yet entirely Python, since I focused on writing the libraries needed for the new approach. Eventually even the IEEE registry archiving will be Python, though that’s a trivial part of the project. The biggest blockade I’d had with the new code was handling Unicode issues thanks to a combination of mixed-character-set data and Python 2.x’s notoriously painful Unicode support.
I hit massive milestone yesteday: The new Python code processed nearly 20 years worth of IEEE registry files without dying once due to one error/bug or another! However, the resulting collection of metadata isn’t accurate due to other more subtle bugs. My next steps will be to:
While all these updates are going on behind the scenes, the old Perl & PHP codebase is still active and running! If you go to http://search.deepmac.org/ you’re accessing the page which used it, including the MySQL database. But the old codebase doesn’t support the newer MA-M and MA-S assignments IEEE created. It also isn’t very efficient.
I hope to have the new codebase more filled out in the next year or so to the point where I (and hopefully others) will be able to use it as a platform to develop a whole new search tool that will allow access to all the metadata.
August 21, 2017
You may have noticed everything looks different. I’ve moved to a new domain, added a Wiki page and re-arranged the main page. But those are only the visible changes.
Internally, I’ve normalized my archive of IEEE registry files (over 10 years worth), cleaned-up the old MySQL database and made significant progress on a re-write using Python. The code is all available on GitHub (see links at top and bottom of this site). The reboot of this project means more than re-writing code, I’ve also re-designed how all the registry files are archived and how all metadata is stored and organized. A new database design will be needed to make this reboot functional as a search page, but the raw JSON data will be exposed in the near future.
July 21, 2017
Komodo Laboratories has released Slitheris Network Discovery 1.0, a Windows application for performing network discovery. This is the first known commercial use of data from the DeepMac project, which Slitheris leverages to provide date approximation for discovered devices.
Announcement page: http://www.komodolabs.com/network-scanner-news/
Direct Download: http://www.komodolabs.com/downloads/slitheris_installer.exe
November 10, 2015
I’ve updated the Perl code used to archive the IEEE registry files for OUI’s (now relabeled by IEEE but that’s solely for branding purposes so I’m not going to bother to switch my own terminology). The changes were mostly to deal with the new size offered (OUI 28-bit) and some minor formatting changes. Only other change was to move to a recursive directory storage for storing the files to keep things neater.
The real work has been on all-new code written in Python to support a new approach for storing DeepMac metadata. The classes I’m working on will eventually allow for adding, deleting and changing data in the repository in a journaling style, where all changes are recorded. So even if something is “deleted” there’s still a record of it having existed. This will be quite handy in the future.
Once the code is actually working with an initial filesystem prototype there will eventually be a Web-based and database-based connection supported. My plan is to have ultimately have an API that will make it programatically simple to be able to add new data to the repository, either manually or in an automated way.
The final step will be to have the current web-based search system moved to using the new repository on the backend (it’s still using the old MySQL database I originally created in the early alphas). The search engine will let you see the “current” snapshot of all the metadata but also allow a view of historical data.
December 2, 2009
Almost a year after I first wrote about my idea for DeepMac, I can finally show something actually functional for the world to point and stare at!
DeepMac ALPHA is available at http://deepmac.jedimercer.com/ and includes the full MySQL dump of the database contents.I’m licensing the database under the Open Database License.
At this point, there’s only a few hundred OUI’s tied to a device of some sort. This is because I’ve had to focus on developing the skeletal framework to actually store the information, and have not yet put together tools for automating any part of the process of submitting data. Still, there’s a good breakdown of virtual computers, network-attached cameras, the major Smartphone models and a few videogame systems.
For DeepMac to become truly useful though, it needs a lot more raw data, and not just the easy pickings. That means there needs to be better ways to get data into DeepMac. A primitive search interface is just a baby-step. Ideally, we need to have a standard format for submitting new DeepMac data for analysis and inclusion. Mostly this is about sitting down and figuring out what would work best.
The other thing DeepMac needs to be useful is of course a real-world application that everyone can benefit from. I’d love to see DeepMac data integrated with great security tools like WireShark, nMap, Kismet and others. There’s also a ton of commercial security software that could benefit from DeepMac such as SIMs, asset management tools and anything else that does network discovery. Alas, I don’t see that happening anytime soon. A side-project of mine is developing a tool for tracking IP<->MAC pairings for network surveillance. I’ll be integrating DeepMac data into that tool, and hopefully releasing it as an Open Source project sometime next year.
In the meantime, I’m still very much interested in feedback about the DeepMac project! What I’ve gotten so far has been small but very useful and greatly appreciated. And if you have any intelligence to offer the DeepMac project (as in OUI Device mapping), please e-mail me!
July 9, 2009
It’s been almost exactly one month since my last entry on DeepMac’s goals and I’m happy to say significant progress has been made!
The biggest achievement is getting the project finally organized, with clear goals, usable data and a skeletal framework. But that’s kind of broad, so let’s breakdown what has been achieved:
Historical creation dates for OUIs have been documented going back to 1998, thanks mostly to archive.org and the magic of Google. Thus the creation of the DeepMac knowledgebase! An automatic archiving of the IEEE OUI listing in text format has been established, running on a daily basis. Simple Perl scripts have been written to convert the IEEE OUI file (oui.txt) into a tab-delimited format, one OUI per line, and to add OUI creation dates from the DeepMac knowledgebase. An actual MySQL database has been created and another Perl script used to load some of the knowledgebase data into the MySQL database. A horribly primitive PHP interface has been written to allow very basic searching of the DeepMac database.
There is still an enormous amount to accomplish with this project. I’ve been thinking about where the focus should be and while I really want to evangelize the project and get input from the big boys in networking, it seems to me that it will never be manageable until the project is fully organized and has the tools it needs. So the push for now is going to be on continuing to refine the MySQL database for DeepMac in terms of structure, and lots of work on a web-based front-end.
Ideally, it should be possible to not only fully search DeepMac on-line but also submit information, and for administrators to update the contents including approving submissions for inclusion. That’s a tall order, so the first sub-goal here will be to whip the search interface into shape and allow full searching on multiple fields with varying levels of output detail.
June 18, 2009
DeepMac has one primary goal, to provide as much intelligence about MAC addresses as possible. While IEEE provides a simple directory of OUI (aka MAC prefix), we want more. Here I’ll outline the milestones for this goal, with the realization that it is unlikely that all OUI assignments will reach the level of detail desired. Creation Dates
I’ve verified with IEEE officials that the assignment date for each OUI is on record. However, that information is not freely available. IEEE officials did inform me that an individual organization may submit a request to find the creation date for a particular OUI assigned to them. Oh, and why do we want creation dates? It can be a very rough indication of the age of a device. For example, a CISCO device with a MAC prefix assigned in the 1990’s is more than likely an elderly model. Device Classification
The real meat of the DeepMac project is to try and classify as many OUI’s as possible by the device the MACs are assigned to. For a very large number of MAC prefixes, these will be network interface cards, but in the past 10 years we’ve seen an explosion of embedded networking in consumer electronics, vehicles, and industrial devices. Device classification will be the most research-intensive aspect of the project, requiring a lot of searching, requests for assistance, and verification via input from device owners. Vendor Participation
It is not likely that DeepMac will succeed in its goals without participation from major vendors such as Cisco, IBM, Apple, and Nokia. Significant blocks of the OUI space are assigned to a relatively small number of companies, and breaking those blocks down for device classification will require “insider” knowledge about how the OUI space was parceled out by the vendor. I intend to reach-out to the security community at large, but also to folks within these large stakeholders. Project Simplification
Most importantly, the DeepMac project can’t succeed if it is not kept simple. There needs to be relatively little effort in order to contribute information. The OUI blocks need to be consolidated for research purposes, so the largest blocks owned by a single company can be focused on. Tools for automatically converting and extracting data related to OUI and MAC prefixes should be developed. And finally, the end-result of all this research and data compilation needs to be a simple, clean data file that can easily be parsed or re-compiled for use by various tools.
So here’s the pitch: I need help! What I’m currently looking for is participation, in the form of people with first-hand knowledge about a particular MAC prefix or company with OUI assignments. I’m also looking for contributions of device classifications. The Cisco OUI space is going to be a big priority, both due to size and the utility of the additional intelligence. So if you work for a major company that owns OUI space, please drop me a line! General suggestions and comments are also welcome, of course.
You may e-mail me at firstname.lastname@example.org