Licensing and readme

.gitignore

@@ -1,3 +1,7 @@
+## Project specifics
+testcases/*/
+
+## C & C++
 # Prerequisites
 *.d
 
@@ -35,7 +39,7 @@
 [Bb]uild/
 cmake-*/
 
-## Python begins here.
+## Python.
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

LICENSE

@@ -1,4 +1,4 @@
-MIT License Copyright (c) <year> <copyright holders>
+MIT License Copyright (c) 2021 Erki Meinberg
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,13 +1,124 @@
-# masters-thesis
+# clang Based Atomic Operations Parser & Analyzer
 
-## Building
+The source code attached for a master's thesis work carried out in 2021.
+Contained within is the source code for an **Atomic Operations Parser** and
+a **Performance Analyzer**.
 
-Conan is recommended. Otherwise you have to provide `Catch2_ROOT`
+## Atomic Operations Parser (Operatioon Finder)
+
+The **Atomic Operations Parser** is a C++ program which is meant to parse C-code
+source files, and extract atomic operations from them. The tool uses LLVM & clang.
+So precompiling the development libraries of those two is required.
+
+### Building LLVM & clang
+
+A docker file which automatically compiles and installs the required dependencies
+is included for convenience.
+
+Otherwise, the steps to installing clang are as follows:
+
+Install the required dependencies via apt:
+```shell
+apt update -y
+apt install -y git build-essential cmake ninja-build python3 python3-pip
+```
+
+Clone LLVM from the repo. Depth 1 makes the process faster. Also set up the various
+folders for building and installing.
+```shell
+cd ~
+git clone --branch "release/11.x" --depth 1 https://github.com/llvm/llvm-project.git
+mkdir ~/llvm-project/build
+mkdir ~/llvm-install
+cd ~/llvm-project/build
+```
+
+Run cmake to configure the project. Followed by ninja to install it.
+Note that when installing, you can modify `-DCMAKE_INSTALL_PREFIX` to specify
+where the libraries should be installed to. In this case, we'll put them into
+`~/llvm-install`.
+```shell
+cmake ../llvm -G "Ninja" -DCMAKE_INSTALL_PREFIX=~/llvm-install -DLLVM_ENABLE_PROJECTS="clang" -DCMAKE_BUILD_TYPE=Release
+ninja install
+```
+
+### Building the Tool
+
+Using the conan package manager is recommended. Otherwise you have to provide `Catch2_ROOT`
 and `nlohmann_json_ROOT` yourself.
 
-```shell
-> mkdir build
-> cd build
-> conan install .. --build=missing
-> cmake .. -DWITH_TESTS=ON -DClang_ROOT=${CLANG_ROOT} -DLLVM_ROOT=${LLVM_ROOT}
+The variables `Clang_ROOT` and `LLVM_ROOT` depend on the previous step. If you installed the libraries
+into your system, then you don't need to specify them. Otherwise, assuming an installation directory of
+`~/llvm_install`, they'd look as follows:
 ```
+-DClang_ROOT=~/llvm_install/lib/cmake/clang/
+-DLLVM_ROOT=~/llvm_install/lib/cmake/llvm/
+```
+
+Now clone this repo and `cd` inside of it. Make a build directory and build the project:
+
+```shell
+mkdir build
+cd build
+conan install .. --build=missing
+export CLANG_ROOT=~/llvm_install/lib/cmake/clang/
+export LLVM_ROOT=~/llvm_install/lib/cmake/llvm/
+cmake .. -DWITH_TESTS=ON -DClang_ROOT=${CLANG_ROOT} -DLLVM_ROOT=${LLVM_ROOT} -GNinja
+ninja
+```
+
+You are now left with `op-finder/op-finder` and `op-finder-tests/op-finder-tests` executables.
+
+### Usage
+
+Use `op-finder --help` for help.
+
+The finder will process multiple source files and output them to a single JSON file. For example:
+
+`op-finder ./source1.c ./source2.c -o=project_opfinder.json`
+
+The above line will take the C-code source files of `source1.c` and `source2.c`, extract atomic operations
+from them, and output the JSON to the `./project_opfinder.json` file. This file can then be given to the
+analyzer along with a gcov report.
+
+## Analyzer (Operation Summarizer)
+
+The Analyzer is responsible for taking the Atomic Operations Finder report and a gcov code coverage report
+and combining them into a singular analysis of the codebase. In the present implementation, it will
+summarize all unique atomic operations. This can then be combined with a database of atomic operations
+and turned into a performance estimation.
+
+The Analyzer is written in Python and requires no tooling beyond having Python 3 installed. gcov is needed
+to generate the simulation reports.
+
+### Prerequisites
+
+Install the prerequisites from your package manager:
+```shell
+apt update -y
+apt install gcov python3 python3-pip
+pip3 install gcovr
+```
+
+### Usage
+
+Assuming Atomic Operations Parser was used in the previous usage example. The first step is to compile
+the C program with GCC and to acquire a code coverage report from it using gcovr. This is done as follows:
+```shell
+gcc -fprofile-arcs -ftest-coverage -fPIC -O0 ./source1.c ./source2.c -o project.out
+./project.out
+gcovr -r ./ --json-pretty -o project_gcov.json
+```
+
+This will output the coverage report in human-readable JSON into the `project_gcov.json` file.
+
+Next, the Analyzer needs to be ran:
+```shell
+python3 op-summarizer/opsummarizer.py --gcov project_gcov.json --finder project_opfinder.json --output project_summarized.json source1.c source2.c
+```
+
+The list of files in the specifies which source files should be taken into consideration. If a file is not present, then
+that file will not be evaluated during the summarization.
+
+The summarizer will then generate an output report in JSON, along with printing a human-readable version out
+on the screen.

op-finder-lib/src/OperationFinderAstVisitor.cpp

@@ -28,8 +28,7 @@ bool OperationFinderAstVisitor::VisitBinaryOperator(clang::BinaryOperator* op)
 {
   assert(_context);
 
-  if (!op->isCompoundAssignmentOp())
-    _op_finder->processArithmetic(op, _context->getSourceManager());
+  _op_finder->processArithmetic(op, _context->getSourceManager());
 
   return true;
 }

op-finder/main.cpp

@@ -72,7 +72,7 @@ int main(int argc, const char** argv)
 
   OperationFinderAstAction action(&op_finder);
 
-  assert(!Tool.run(newFrontendActionFactory(&action).get()));
+  Tool.run(newFrontendActionFactory(&action).get());
 
   if (!OutputFile.getValue().empty())
     storage.toFile(OutputFile.getValue());

op-summarizer/gcovreader.py

@@ -27,13 +27,13 @@ class GCovFile:
             lines: List[GCovLine] = []
 
             for line in file["lines"]:
+                # Branch specific identification. TODO! later.
+                lines.append(GCovLine(line["line_number"], line["count"]))
                 if len(line["branches"]):
                     # GCov reports branches in reverse order to our parser.
                     branch_number = len(line["branches"])
                     for branch in line["branches"]:
                         lines.append(GCovLine(line["line_number"], branch["count"], branch_number))
                         branch_number -= 1
-                else:
-                    lines.append(GCovLine(line["line_number"], line["count"]))
 
             self.files[name] = lines

op-summarizer/opsummarizer.py

@@ -17,6 +17,8 @@ class OpSummarizer:
 
     def count_operations(self, file: str) -> Dict[UniqueOperation, int]:
         if file not in self.gcov.files or file not in self.ops.files:
+            print(f"Gcov files: {self.gcov.files.keys()}")
+            print(f"Opfinder files: {self.ops.files.keys()}")
             raise RuntimeError(f"File {file} not in both parsers.")
 
         op_counter: Dict[UniqueOperation, int] = {}
@@ -67,6 +69,7 @@ if __name__ == "__main__":
     summarizer = OpSummarizer(args.gcov, args.finder)
 
     total_count = {}
+    total_num = 0
 
     for file_name in args.files:
         ops = summarizer.count_operations(file_name)
@@ -75,9 +78,12 @@ if __name__ == "__main__":
         print(f"Unique operations for file {file_name}:")
         for uop, count in ops.items():
             print(f"\t{count}: {uop}")
+            total_num += count
 
         print("---------")
 
+    print(f"Total count: {total_num}")
+
     if args.output:
         with open(args.output, "w") as outfile:
             json.dump(total_count, outfile)