Compiling Proto Files in Third-Party Jars with Maven

Introduction

When working with third-party Java Archives (JARs) that include Protocol Buffer (protobuf) definitions, we often run into the challenge of reusing those .proto files in our own project. Proto files define the structure of your data and are compiled to generate source code that you use to write and read your structured data.

The problem arises because these .proto files are typically not compiled into the JARs, so we need to extract and compile them separately to use them in our own codebase. Compiling these files is essential because it turns the high-level syntax defined in the .proto files into language-specific code that our application can use.

Without a proper process, we may try to manually extract these .proto files, which is time-consuming and error-prone. This is where Maven—a powerful build tool—comes into play. Maven can be configured to automate the extraction and compilation of .proto files from third-party JARs, which significantly streamlines the development process.

In the following sections, we will explore how to configure Maven to handle .proto files from these third-party JARs and ensure a smooth integration into our build process.

Maven Configuration

Extracting .proto Files from JARs

To use the .proto files in our project, we need to extract them from the third-party JARs. The following Maven configuration automates this process:

<build>
    <plugins>
        <!-- Add the dependency plugin -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>3.1.2</version>
            <executions>
                <!-- Unpack execution -->
                <execution>
                    <id>unpack-protos</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>unpack</goal>
                    </goals>
                    <configuration>
                        <artifactItems>
                            <!-- Specify the third-party JAR containing the .proto files -->
                            <artifactItem>
                                <groupId>third.party</groupId>
                                <artifactId>third-party-library</artifactId>
                                <version>1.0.0</version>
                                <type>jar</type>
                                <overWrite>false</overWrite>
                                <outputDirectory>${project.build.directory}/protos</outputDirectory>
                                <includes>**/*.proto</includes>
                            </artifactItem>
                        </artifactItems>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <!-- Protobuf Maven Plugin configuration from previous step -->
    </plugins>
</build>

Make sure to replace third.party, third-party-library, and 1.0.0 with the actual groupId, artifactId, and version for the third-party JAR.

This configuration block will unpack all .proto files from the specified third-party JAR into the target/protos directory during the generate-sources phase of the Maven build.

If you don’t want to extrat all .proto files to the target directory, you can modify <includes>**/*.proto</includes> to include the file(s) you want. For example, <includes>**/names.proto, **/places.proto</includes> tells the plugin to only copy files whose name is names.proto or places.proto to build directory.

Dependencies

To handle protobuf files, we need to add the necessary dependencies to the Maven pom.xml file. These dependencies provide the required libraries to work with Protocol Buffers in your project.

<dependencies>
    <!-- protobuf-java dependency for Protocol Buffers Java API -->
    <dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <version>4.27.0</version>
    </dependency>
</dependencies>

Make sure to replace 4.27.0 with the latest version compatible with your project.

Plugins

Next, we’ll add the protobuf Maven plugin, which invokes the Protocol Buffers compiler to generate Java source files from .proto files.

<build>
    <extensions>
    <!-- automatically generate a classifier for the current OS and architecture -->
      <extension>
        <groupId>kr.motd.maven</groupId>
        <artifactId>os-maven-plugin</artifactId>
        <version>1.6.0</version>
      </extension>
    </extensions>
    <plugins>
        <!-- Protobuf Maven Plugin configuration -->
        <plugin>
            <groupId>org.xolstice.maven.plugins</groupId>
            <artifactId>protobuf-maven-plugin</artifactId>
            <version>0.6.1</version>
            <configuration>
                <protocArtifact>
                    com.google.protobuf:protoc:3.4.0:exe:${os.detected.classifier}
                </protocArtifact>
                <!-- Add additional configuration as required -->
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>test-compile</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <!-- Add other necessary plugins -->
    </plugins>
</build>

Again, make sure the versions of the protobuf-maven-plugin and protoc match the version of the protobuf-java dependency and that they are compatible with each other.

It is recommended to use os-maven-plugin to automatically generate a classifier for the current OS and architecture. This plugin is also used by protobuf team at Google to generate classifiers for protoc artifacts.

This plugin configuration ties the Protocol Buffers compiler into the Maven build process. It will automatically generate Java source files each time the project is built.

Once the .proto files are extracted into target/protos, the protobuf Maven plugin we configured will automatically pick them up and compile them. If necessary, you may need to adjust the protoSourceRoot within the protobuf plugin configuration to point to your target/protos directory:

<!-- Inside protobuf-maven-plugin configuration block -->
<configuration>
    ...
    <protoSourceRoot>${project.build.directory}/protos</protoSourceRoot>
    ...
</configuration>

If you only want to compile some, not all .proto files, includes the file you want. For example:

<!-- Inside protobuf-maven-plugin configuration block -->
<configuration>
    ...
    <includes>
        <include>example/common/config.proto<include>
    </includes>
    ...
</configuration>

Make sure the .proto files that are imported (used) by your file are also included.

Running the Maven build now (mvn compile), the plugin should compile the .proto files and generate the Java classes in your target/generated-sources directory.

(Optional) Integration with your IDE

After the Protocol Buffer files have been extracted from the third-party JARs and compiled, the generated source files need to be integrated into your project’s build path. Maven handles this automatically when it generates the code in the target/generated-sources directory.

However, some IDEs won’t recognize it. To ensure that your IDE recognizes these sources and includes them during compilation, you might need to add the following snippet to your pom.xml:

<build>
    <!-- Add the following to the existing build configuration -->
    <plugins>
        <plugin>
            <!-- Build Helper Maven Plugin to add source directories -->
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>build-helper-maven-plugin</artifactId>
            <version>3.2.0</version>
            <executions>
                <execution>
                    <id>add-source</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>add-source</goal>
                    </goals>
                    <configuration>
                        <sources>
                            <source>${project.build.directory}/generated-sources</source>
                        </sources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

This configuration will make the Maven Build Helper Plugin add the generated sources to your project’s source directories during the build process, meaning compiled .proto files will be included just like your hand-written source files.

Troubleshooting

Encountering issues is common during the build and integration process, here are a few troubleshooting tips:

• If classes generated from the .proto files are not found, check that the protobuf plugin's protoSourceRoot is correctly set and pointing to the unpacked .proto files.
• In case of version conflicts with protobuf or plugin dependencies, ensure that all dependency versions in your pom.xml are compatible.
• If Maven does not generate source code after running the compile command, check for errors during the generate-sources phase and ensure that the path to the .proto files is correct.

Conclusion

We have covered the essential steps to handle Protocol Buffer files in third-party Java Archives using Maven. This includes adding dependencies, configuring the protobuf Maven plugin, extracting .proto files from JARs, compiling them, and integrating the generated classes into your project.

By automating these steps with Maven, you can streamline the process and avoid manual intervention, which makes your build process more reliable and easier to maintain. Additionally, troubleshooting tips help you navigate common pitfalls that may arise.

Thank you for reading, and until next time, happy coding!