Machine Learning on Source Code

Egor Bulychev

@egor_bu

#MLonCode

Egor Bulychev

@egor_bu

🚀 Machine Learning

GitHub

Machine Learning on Source Code

MLonCode

Examples

Code naturalness

class ???:
    def connect(self, dbname, user, password, host, port):
        # ...
    def query(self, sql):
        # ...
    def close(self):
        # ...

class Database:
            def connect(self, dbname, user, password, host, port):
                # ...
            def query(self, sql):
                # ...
            def close(self):
                # ...

class Foo:
            def bar(self, qux):
                # ...
            def baz(self, waldo):
                # ...
            def do(self, really):
                # ...

Projects similar to MariaDB/server?

Details

Exploratory search

Similar code detection

By style
By structure
By identifiers

provides us

💪Global graph
Licenses
Refactoring

Code autocompletion

Code style

class foobar:
    def connecttoserver(self):
        myserverhost = globalconfig.server.host

class FooBar:
    def connect_to_server(self):
        myServerHost = globalConfig.server.host

Many other applications

Prediction of class, function, variable names
Type inference
Which comments do not make sense?
Which comments are funny?
Which APIs are bad or misused?

Your code as a crime scene

Idea: mine the development history
Book by Adam Tornhill
codescene.io

Data

Datasets

GHTorrent - everything except Git repositories, 70GB
Public Git Archive - only Git repositories, 3TB
GitHub Data in Google BigQuery
rovers & borges - DIY

PGA

270k of siva files
CSV index

Details in the paper.

Tools for MLonCode

Plumbing

source{d} engine

siva or bare git repository loader for Apache Spark
Classification, parsing (thanks to bblfsh), filtering, checkouting, scaling
Apache license

GitHub

source{d} engine

>>> from sourced.engine import Engine
>>> engine = Engine(spark, "/path/to/siva/files", "siva")
>>> engine.repositories.references.head_ref \
    .commits.tree_entries.blobs \
    .classify_languages() \
    .select("blob_id", "path", "lang") \
    .show()

AST

How to parse

Regular expressions - Pygments, highlight.js
Abstract syntax tree (AST) - ANTLR
Compilation

Universal AST

± uniform structure
± standard node types (roles)
XPath queries
4 traversal orders

dashboard.bblf.sh

>>> engine.repositories.references.head_ref \
    .commits.tree_entries.blobs \
    .classify_languages() \
    .filter('lang = "Python"') \
    .extract_uasts() \
    .query_uast('//*[@roleIdentifier]') \
    .extract_tokens("result", "tokens") \
    .select("blob_id", "path", "tokens")

Powerful

modelforge

Automatically fetch from the internet
"Model Store"
Flexible, modern binary format - ASDF
Versioning
Reproducability
Support various programming languages

GitHub

Machine Learning

hercules

Go
Mining Git history
Relies on go-git, Babelfish and Tensorflow
Apache license

GitHub

MLonCode in hercules

Structural hotness
Comment sentiment

source{d} ml

Python
Targets large-scale MLonCode
Runs on top of source{d} engine, modelforge and Tensorflow
Apache license

GitHub

Solved practical problems

Identifier splitting
O(1) code similarity search
Topic modeling
Embeddings

Identifier embeddings

$ \begin{split} V_1 \Leftrightarrow & \,\texttt{"foo"} \\ \\ V_2 \Leftrightarrow & \,\texttt{"bar"} \\ \\ V_3 \Leftrightarrow & \,\texttt{"integrate"} \end{split} $

$ distance(V_1, V_2) < distance(V_1, V_3) $

$ distance(V_i, V_j) = \arccos \frac{V_i \cdot V_j}{\left\lVert V_i \right\rVert \left\lVert V_j \right\rVert} $

Scalar product

Norm

How to estimate $V_i \cdot V_j$ ?

class Database:
    def connect(self, user, password, host, port):
        self._tcp_socket_connect(host, port)
        try:
            self._authenticate(user, password)
        except AuthenticationError as e:
            self.socket.close()
            raise e from None

Splitting and normalization

            _tcp_socket_connect -> [tcp, socket, connect]
            AuthenticationError -> [authentication, error]
            authentication, authenticate -> authenticate

class Database:
    def connect(self, user, password, host, port):
        self._tcp_socket_connect(host, port)
        try:
            self._authenticate(user, password)
        except AuthenticationError as e:
            self.socket.close()
            raise e from None

database, connect², user², password², host², port², tcp, socket², authenticate², error, close

class Database:
    def connect(self, user, password, host, port):
        self._tcp_socket_connect(host, port)
        try:
            self._authenticate(user, password)
        except AuthenticationError as e:
            self.socket.close()
            raise e from None

connect², user², password², host², port², tcp, socket², authenticate², error, close

class Database:
    def connect(self, user, password, host, port):
        self._tcp_socket_connect(host, port)
        try:
            self._authenticate(user, password)
        except AuthenticationError as e:
            self.socket.close()
            raise e from None

connect, user, password, host, port

class Database:
    def connect(self, user, password, host, port):
        self._tcp_socket_connect(host, port)
        try:
            self._authenticate(user, password)
        except AuthenticationError as e:
            self.socket.close()
            raise e from None

tcp, socket, connect, host, port

class Database:
    def connect(self, user, password, host, port):
        self._tcp_socket_connect(host, port)
        try:
            self._authenticate(user, password)
        except AuthenticationError as e:
            self.socket.close()
            raise e from None

authenticate², user, password, error, socket, close

class Database:
    def connect(self, user, password, host, port):
        self._tcp_socket_connect(host, port)
        try:
            self._authenticate(user, password)
        except AuthenticationError as e:
            self.socket.close()
            raise e from None

authenticate, user, password

class Database:
    def connect(self, user, password, host, port):
        self._tcp_socket_connect(host, port)
        try:
            self._authenticate(user, password)
        except AuthenticationError as e:
            self.socket.close()
            raise e from None

authenticate, error, socket, close

class Database:
    def connect(self, user, password, host, port):
        self._tcp_socket_connect(host, port)
        try:
            self._authenticate(user, password)
        except AuthenticationError as e:
            self.socket.close()
            raise e from None

authenticate, error

class Database:
    def connect(self, user, password, host, port):
        self._tcp_socket_connect(host, port)
        try:
            self._authenticate(user, password)
        except AuthenticationError as e:
            self.socket.close()
            raise e from None

socket, close

Incidence matrix $C_{ij}$

$C_{ij} =$ number of times $i$ and $j$ were together
Also known as the co-occurrence matrix

Pointwise Mutual Information (PMI)

$$ V_i \cdot V_j = PMI_{ij} = \log\frac{C_{ij} \sum C}{\sum_{k = 1}^N C_{ik}\sum_{k = 1}^N C_{jk}} $$

Representation Learning on Explicit Matrix

Stochastic Gradient Descent

Swivel

Multi-GPU
Multi-node
Quality tricks
Swivel: Improving Embeddings by Noticing What's Missing by Shazeer et.al.
Tensorflow implementation

Nearest to “foo”

afoo
qux
myfoo
baz
mfoo
wibble
dofoo
quux
dfoo
testing
ifoo

Analogies

“bug” - “test” + “expect” = “suppress”

“database” - “query” + “tune” = “settings”

“send” - “receive” + “pop” = “push”

Typos

recieve = receive
grey = gray
calback = callbak = callback

Summary

MLonCode is fun
There is data
There are tools
Community is forming

Idea ? Doubt ? Write us !

Thank you

bit.ly/2MAU7qG

Machine Learning on Source Code

#MLonCode

🚀 Machine Learning

GitHub

Machine Learning on Source Code

MLonCode

Examples

Code naturalness

Projects similar to MariaDB/server?

Exploratory search

Similar code detection

provides us

Code autocompletion

Code style

Many other applications

Your code as a crime scene

Data

Datasets

PGA

Tools for MLonCode

Plumbing

source{d} engine

source{d} engine

AST

How to parse

Universal AST

Powerful

modelforge

Machine Learning

hercules

MLonCode in hercules

source{d} ml

Solved practical problems

Identifier embeddings

How to estimate \(V_i \cdot V_j\) ?

Splitting and normalization

Incidence matrix \(C_{ij}\)

Pointwise Mutual Information (PMI)

Representation Learning on Explicit Matrix

Stochastic Gradient Descent

Swivel

Nearest to “foo”

Analogies

Typos

Summary

Summary

Thank you